sandbox-bench

1---2name: sandbox-bench3description: >4  Benchmark React or Next.js changes on Vercel Sandbox VMs with paired5  A/B statistics: react PR/commit vs base, or Next.js PR/commit vs base,6  measured end-to-end through the bench/render-pipeline app (rps,7  latency, p95; TTFB, RSS and document/Flight bytes when the Next8  side captures them) and, for React changes, through the react9  repo's flight-ssr-bench fixture (Node AND Edge web-streams paths,10  Fizz and Flight+Fizz). Use whenever the user asks to bench, perf11  test, or A/B a React PR, a react-server-dom / Flight / vendored React12  change, or a Next.js PR ("is this PR faster", "does this regress13  RSC?", "measure the perf impact of <commit>"), even if they don't say14  "benchmark" — any request to quantify a server-side performance15  difference between two revisions belongs here. Runs remotely16  (laptop-free), applies correctness gates before measuring, and17  reports boot-level confidence intervals.18metadata:19  internal: true20---21 22# Sandbox bench: paired A/B perf runs for React and Next.js changes23 24Measures what a change is actually worth, end to end: two revisions25("arms") built into otherwise-identical Next.js apps, exercised by the26`bench/render-pipeline` harness on Vercel Sandbox VMs, compared with27paired statistics that treat the VM boot as the unit of replication.28All heavy work happens on sandbox VMs; the laptop only orchestrates.29 30Scripts live in `scripts/` next to this file and run from anywhere.31Arms are git refs, resolved in cached clones of react and next.js;32the Next side defaults to canary. Everything is cached33content-addressed: first use of a new pair builds caches (~45-60 min34extra, once); later runs boot straight into measurement.35 36## One-time setup37 381. `node scripts/config.mjs show` — if it reports NOT CONFIGURED, ask39   the user which Vercel **team** and **project** the sandbox VMs40   should run under (these are billed resources; never guess, never41   default), then `node scripts/config.mjs set team=<slug> project=<name>`.42   Config lives in `~/.config/sandbox-bench/config.json` — never commit43   team/project names into the repo.442. The Vercel CLI session must have access to that team. On a 403,45   stop launching (don't retry through it) and check whether access is46   already back: `vercel whoami --scope <team-slug>` plus one scoped47   read call (e.g. `vercel sandbox ls`) — grants drop and recover on48   their own, and a transient 403 needs no login at all. If49   verification still fails, run `vercel login <team-slug>` yourself50   as a background task (the token lives with the CLI session, not51   with the user). It opens a browser/device confirmation — relay the52   URL if one is printed — but keep re-running the verification pair53   every minute or two while it waits: access often returns before54   the login flow reports success, and once verification passes, kill55   the pending login and resume. After a 403 outage, expect in-flight runs to have died:56   run `node scripts/bench-status.mjs` and follow its recovery57   actions (measurement VMs will have hit their ~5h timeout if the58   outage was long — those cells need relaunching, not collecting).593. react and next.js clones land in the cache on first use (or point60   `reactRepo`/`nextRepo` in the config at existing checkouts).61 62Before the first real run with a new configuration, sanity-check the63plan with `--dry-run` (prints what would happen, touches nothing).64 65## Workflow66 67### 1. Resolve what's being compared68 69- **React PR**: `--pr <url|number>` — base is computed automatically70  (merge-base of the PR head with react main).71- **React refs**: `--arms base=<ref>,cand=<ref>` — base FIRST. For a72  multi-commit branch, base is the merge-base with main, not `cand^`.73- **Next.js PR**: `--next-pr <url|number>`. The React side defaults to74  whatever each Next ref vendors (that's what would ship); pass75  `--react-ref` only to pin both arms to one specific React build.76- **Next refs**: `--next-arms base=<ref>,cand=<ref>`.77 78Exactly one side varies; the other is identical in both arms. That79isolation is what makes the numbers attributable — never vary both.80 81### 2. Gate correctness before spending bench compute82 83A bench number from an arm that fails its own tests is meaningless.84For any arm that is not already CI-green upstream (hand-assembled85branches, cherry-picks with resolved conflicts, local commits):86 87```sh88node scripts/sandbox-gate.mjs --arms cand=<ref>89```90 91The bench itself enforces the primary gate: every react arm's commit92must have green CI on the react repo, checked automatically before any93build or VM is spent. PRs and main-history commits normally satisfy94this with no extra work. For local or unpushed refs (no CI exists),95gate on a VM with sandbox-gate.mjs and then pass --allow-ungated to96the bench. The VM gate runs the full test suite in prod mode (the97channel that gets benched). PASS requires seeing the actual test98counts in the output. If a gate fails, report the failures and stop —99do not bench a broken arm. Each arm is gated in its own100lockfile's environment. Bench the exact sha the gate prints (a branch101ref can move between gate and bench).102 103### 3. Launch the bench (background, non-blocking)104 105```sh106bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \107  2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'108```109 110For React PRs, launch BOTH suites (separate background tasks; they111share arm builds and caches):112 113```sh114bash -c 'node scripts/sandbox-ssr.mjs --pr <url> --label <slug>-ssr \115  2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'116```117 118The e2e suite measures the Node path through a real Next.js app; the119ssr suite measures the react repo's flight-ssr-bench fixture — 8120variants (Fizz and Flight+Fizz, Node and Edge web streams, sync and121async), each sequentially with Flight script injection and behind an122HTTP server at c=1/c=10. Edge cells are the ssr suite's headline (the123e2e suite cannot see that path); its Node and Fizz-only cells124attribute an effect to the Flight layer, the Fizz layer, or the125stream plumbing. The fixture (the workload) is pinned to one ref for126both arms — react main by default — so only the React builds differ;127if the PR itself edits the fixture, the launcher says so and the run128does not measure those edits. Next PRs run the e2e suite only.129 130- Run it as a background task and proceed on its completion131  notification. Never hold a foreground wait; never poll in a loop —132  the rule is about control flow, not status relay: reading the133  output tail to answer "how's it going" is always fine.134- The harness handles the invariants internally: both arms in the same135  VM, interleaved ABBA, paired per (vm, run); detached remote136  execution (transport drops don't kill runs); build fingerprints137  recorded in every result row.138- `live ...` lines are streaming estimates for progress display only.139  Never stop a run early because a live p-value looks good, and never140  report a live number — sequential peeking manufactures false141  positives. Only the final analysis counts.142- Defaults (16 VMs × 2 paired runs) implement the methodology; don't143  reduce VM count to save time — boots are the unit of inference, and144  fewer boots means wider intervals, not faster answers.145- Sandbox compute is internal capacity, not a budget: launch, relaunch,146  and confirm runs without asking about cost or shrinking them to147  save it.148- The Next side's default, `canary`, is the latest published canary149  release (the launcher prints its version and sha), so repeat benches150  reuse the built snapshot until a new canary ships.151- Useful flags: `--bench-env KEY=VALUE` (runtime-only env for the152  bench process — it does NOT affect the snapshot's app build), `--isolate-routes`153  (tail investigations), `--no-profile` (skip the CPU capture that154  runs by default after the timed runs), `--prepare` (build caches155  only — use when two cells will share an arm, to avoid duplicate156  builds racing).157- CPU profiles are captured by default: one profile pass per arm runs158  strictly AFTER the timed runs (it cannot touch the numbers), costs159  ~45-60 min extra VM wall-clock, and lands in `<runDir>/prof-vm<N>/`160  as standard V8 `.cpuprofile` files. Cross-VM profile diffs are161  highly stable (observed 16/16 sign agreement on real movers), so one162  profiled cell suffices to rank hot paths. Analysis caveats:163  aggregate by (functionName, line, column) — bare minified names164  collide across the bundle — and never diff arms by minified name165  (the minifier renames between builds); match positions or code166  snippets instead.167- The bench exercises Next's node-streams path168  (`__NEXT_USE_NODE_STREAMS` is inlined as true for the node runtime169  at build time). React changes that only touch the EDGE stream170  configs are not exercised end-to-end and will (correctly) bench as171  no detected difference.172 173### 4. Read the result like a skeptical data scientist174 175The goal is the truth about the change, not making its author feel176good. The final analysis prints, per route/phase/metric, the177boot-level mean, ±95% CI, and p across boots. Apply the policy in178[references/methodology.md](references/methodology.md):179 180- Claim only boot-level p < 0.01, with the CI, on an A/A-validated181  team/config (see methodology).182- The PR is a hypothesis, not an explanation. Claims come from the183  analysis output alone. When the numbers agree with the PR's story,184  check whether the captured data actually discriminates that185  mechanism from alternatives — a latency win attributed to smaller186  payloads should come with a document-bytes delta; if the bytes187  didn't move, the story doesn't hold and the report says so.188- Use every captured metric, and voice anything that does not add up:189  one metric family moving against the others, effects with no190  byte-level or RSS trace, throughput moving without latency,191  sign flips across boots. An inconsistency you cannot explain192  belongs in the report, not in the drawer.193- The `within-run p` shown in brackets is a diagnostic, never a claim.194- Check the fingerprint header first: two distinct fingerprints = valid195  A/B; "inconsistent fingerprints" = invalid, report no numbers. The196  fingerprint hashes both bundlers' compiled server files — arms197  touching only client files can still legitimately show identical198  fingerprints with different version strings.199- Per-boot values are printed; if boots disagree in sign, say so.200- Any claim that will drive a decision gets one independent201  confirmation run before it's stated as fact.202 203Re-analyze any past run without re-running it:204`node scripts/bench-analyze.mjs <runDir>`.205 206### 5. Report207 208Name what was measured with links: the PR title (printed in the209analysis header, stored in meta.json) linking to the PR; for ref210arms, the commit title. Lead with a table of the significant cells,211each row carrying the effect with its unit, the CI, and p:212 213```214## [<PR title>](<PR url>) — e2e, Vercel Sandbox (x86 Xeon), <n> boots215 216Significant (boot-level p < 0.01, A/A-validated):217| cell | effect | 95% CI | p |218|---|---|---|---|219| /dashboard under load | +14.4% throughput (req/s) | ±3.2% | <0.0001 |220| /dashboard serial | −10.7% median latency (ms) | ±0.6% | <0.0001 |221 222No detected difference: <every cell not in the table, by name>.223Flags: <cells at 0.01 ≤ p < 0.05, sign disagreements across boots,224fingerprint caveats, anything that does not add up>225```226 227One row per cell: rps and median restate each other, so report the228throughput number (add a p95 row only when the tail moves differently229from the median). Document metrics (raw/gzip/Flight KB) get their own230rows when they differ — they are the mechanism evidence. When the231Next side predates the document-metrics harness (vercel/next.js#95828)232those cells are absent; say so instead of silently reporting less. State the platform next to the numbers. Magnitudes233are platform-dependent (GC share differs by CPU); direction and234mechanism transfer, percentages do not. Never present a235noise-compatible delta as a small win or loss — it is "no detected236difference".237 238## Results database239 240Every collected run lands in one SQLite file,241`~/.cache/sandbox-bench/results.db` — raw measurements and artifacts242(CPU profiles, logs) only, written exclusively by the importer, never243by hand. The launcher imports and verifies automatically at244collection; `bench-analyze` reads the db and nothing else, so every245statistic is a pure function of it. Numbers in reports come from the246analysis output verbatim — never retype, recompute, or aggregate them247yourself.248 249- `node scripts/bench-db.mjs ls` — all runs with sample/artifact counts.250- `node scripts/bench-db.mjs verify [runId]` — integrity checks:251  sqlite-level, referential, one fingerprint per arm, paired sample252  counts, artifact sha256. Run it before drawing on old data.253- `node scripts/bench-db.mjs export out.db <runId...>` — cut a254  self-contained db of specific runs (with their profiles) to send to255  someone. It opens in any SQLite tool.256- `node scripts/bench-analyze.mjs <runId>` — re-analyze anything in257  the db; a run-dir argument imports it first.258 259## Keeping the user informed260 261The launcher narrates itself on stdout: launch facts first (run dir,262arms, CI verdicts), then a progress line every ~2 minutes with rows263collected and interim per-route effects with confidence. Relay to the264user: the run dir and expected duration right after launching,265notable interim shifts if they ask how it's going, and the full266verdict from the final analysis when the completion notification267arrives. The analysis names metrics that were `not captured on this268run` — repeat that in the verdict when it limits what the data can269say (document metrics absent means the payload mechanism is270unverified, not verified-identical).271 272While a run is active, open any reply with a one-line status per run:273read the tail of the launcher's output and quote its latest progress274line. If the session supports timed wakeups or reminders, schedule a275check at each expected transition (arm builds -> experiment snapshot276-> measuring, then every ~15 minutes of measurement) and post the277progress line; if not, say when the next update will arrive so278silence is never ambiguous. Interim effects in progress lines are279streaming estimates — share them as progress, never as claims.280 281If a launcher process dies (session teardown, crash), the remote VMs282keep executing their measurement loops — the data is not lost. `node283scripts/bench-collect.mjs <runDir>` reconnects, waits for the loops,284downloads the results, cleans up, and analyzes. Run it before the VMs285hit their ~5h timeout.286 287## Failure recovery288 289- **First move, always: `node scripts/bench-status.mjs`.** Session290  restarts silently kill background launchers while their detached VMs291  keep measuring, and a dead launcher's log still ends with a292  healthy-looking progress line — never infer liveness from log tails293  or task output files. bench-status checks each run's recorded294  launcher pid and prints the per-run recovery action (running /295  collect now / relaunch). Run it at the start of any session that296  expects work in flight, after any crash, and before telling the user297  what is or isn't running. Launcher crashes are also recorded in the298  run's status.json (`phase: "failed"` plus the error).299- **Interrupted local process**: remote VMs keep running detached.300  `vercel sandbox list` (with the configured team/project) to find301  them; poll each VM's `/vercel/sandbox/loop.done`, `cp` its302  `results.jsonl` down when done, then remove the VM and analyze with303  `bench-analyze.mjs`.304- **Leaked VMs** after any crash: `node scripts/sandbox-sweep.mjs`305  lists this skill's VMs (matched by sbench-\* name AND the306  purpose=sandbox-bench tag, and only when older than --min-age-hours,307  default 3, so healthy in-flight runs are never touched); `--yes`308  removes them by exact listed name.309- **Flaky uploads/transports**: the harnesses size-check artifacts and310  abort on truncation. A failed cell is safe to relaunch; caches make311  the retry cheap. Don't relaunch two cells that need the same uncached312  arm at the same moment — they'll race to build it; use `--prepare`313  first instead.314 315## Cost expectations (set these with the user before big runs)316 317Per cell at defaults: ~18 VMs (8 measurement + build/snapshot VMs),318~1-2h wall-clock cold, ~30-60 min warm. A/A calibration and319confirmation runs are extra cells. VMs are billed to the configured320team — for anything beyond a single PR check, confirm scope first.321