Your benchmark runs are not independent samples

This one is for the re-run @yohjisakamoto described: raw provider usage objects, normalized fresh/cache-read/cache-write splits, measured vs estimated buckets, n≥5 with bootstrap CIs and tier stratification, wiring and meter checks as preconditions. That list is right, and it's more rigor than any public comparison of these tools has had. But every gate in it validates a single run's record. Nothing in it validates the sample — and the statistics you're about to compute assume something about the sample that agentic benchmark runs quietly violate.

A bootstrap CI treats your runs as independent draws from one distribution. In this domain they aren't, in at least five specific ways. Each of these cost us a wrong conclusion (or nearly did) before we found it, and none of them appear in any methodology writeup we've seen. Consider this the field guide we wish someone had handed us.

1. The cache couples consecutive runs

Provider prompt caches persist for minutes to an hour after a request. Run configuration A, then configuration B on the same task, and B inherits whatever prefix cache A just paid to write. Whichever arm runs second looks cheaper — regardless of the tool.

This is not a small term. In our own editor, cache state alone moved a session's bill ~6× with zero change to model, tool, or prompt (the cache-churn incident from the earlier post: 4% read rate where ~95% was expected, $0.59 of a $0.60 session in cache writes). When 96% of tokens are cached input, cache state is a bigger term than most of the tool effects anyone is hunting. A comparison that doesn't control it is measuring cache weather.

What we do about it:

• Interleave arms, never run them in blocks. A-A-A-A-B-B-B-B maximally confounds arm with cache state (and with time — see §2). Alternate or randomize.
• Decide a cold-start policy and record it. Either force cold (gaps longer than the cache TTL, or a distinct prefix per run) or give both arms the same warmup and drop run #1 from each.
• Keep fresh-input as its own column. With the read/write/fresh split recorded per run, you can compute the comparison both on total billed tokens and on fresh input alone. When those two disagree, cache coupling is the reason — and that disagreement is itself a finding.

2. A sweep is a time series, and the system drifts under it

An n≥5-per-cell, tier-stratified sweep takes days. Providers ship silent model updates on that timescale. Cursor ships harness updates on that timescale. Pricing tables drift on that timescale — our own cost readout was confidently 7× wrong for a while, and that was client-side drift; the provider-side kind you can't even read in a diff.

If arm A ran Monday and arm B ran Thursday, you compared two different worlds and labeled the difference "the tool."

What we do about it:

• Our measurement harness interleaves per step — each step is a complete model×variant comparison, so drift lands on both arms equally instead of on one.
• Record the model ID from the response, not the request. The request says what you asked for; the response says what actually served the run.
• Record harness and plugin versions in the run row. If either changes mid-sweep, the sweep has a seam — analyze the halves separately or start over.

3. Your sample is a mixture, and the mean of a mixture lies

Our noisiest model refused to produce a stable token number: a 6.4× range (4,091 to 26,358 tokens) on the same task, CV ~50%. The temptation is to call that randomness and average harder. It wasn't randomness. Reading the logged commands showed a distinct failure mode — the model used the terminal as its primary interface, spelunking the shell (cat, sed, find, grep, re-running the same build) instead of using its tools, and how deep it dug was the variance.

That means the sample wasn't one distribution. It was a mixture: clean runs and spelunking runs. And interventions act on the mixture, not the mean — the directive we shipped for this cut the mean −38% (n=12), but its real effect was killing the catastrophic tail: worst case dropped 47%, from 34,154 to 18,159 tokens.

What we do about it:

• Classify every run before averaging: completed clean / wandered / retried / failed. The classification comes from the logs, and it's cheap.
• Report the mixture shift next to the mean shift. A tool that converts wanderers into clean runs can leave the median untouched and still be the most valuable thing you tested. Averaged blindly, it looks like nothing.
• Work in log space (multiplicative costs, heavy tails) — the log-ratio bootstrap from earlier in this thread exists because one 26k-token run should not own your mean.

4. Variance is a result, not an error bar

The most robust finding in our whole efficiency investigation wasn't a mean. On one reasoning model, our intervention read −32% mean tokens — but the number that replicated cleanly was the variance collapse: CV 75% → 22%. Runs went from swinging 6× to landing in a tight band. (An earlier n=2 probe of the same effect read −63%; the direction was robust, the magnitude was n-dependent. Treat your own early magnitudes the same way.)

Two tools with identical means and 3× different spread are not equivalent products. The tight one you can budget; the wide one hands you a surprise bill some fraction of the time. If the re-run reports only means and CIs, this dimension is invisible.

Report per-arm CV (or IQR) as a first-class result. A variance change with a flat mean is worth publishing on its own.

5. The verifier is part of the instrument

Billed-usage-per-verified-success puts a verifier inside the measurement loop, and the verifier has its own failure modes. Two of ours:

• Metric mixing. Our headline eval score climbed impressively — and decomposition showed it climbed because apply-rate climbed (did the edit apply at all), not correctness. One score was silently blending "produced output in the right shape" with "produced the right output." De-mix before you trust a denominator: applied? passed real tests? honest about what it did?
• The too-good score. A 100% result turned out to be train/eval leakage — the system had effectively seen the answers. Your version of this is subtler: repeated runs of the same task pool, with caching and retrieval in the loop, can "learn" the benchmark across repetitions. A score that improves across repeats of an unchanged configuration is a leak detector firing.

And if verification is an LLM judge: pin its version, record it per run, and accept that its variance is now inside your CI. Given the coupling point from earlier in this thread (compression moves usage and pass rate), a drifting verifier makes any usage delta unfalsifiable — you can no longer say whether the bill moved or the bar did.

What this adds to the run row

The record schema this thread converged on, plus the sample-level columns:

• interleave position / arm order index, and timestamps
• cold/warm cache flag (and the fresh-vs-total comparison both ways)
• model ID as reported by the response, harness + plugin versions
• run classification (clean / wandered / retried / failed), from logs
• verifier ID + version, with de-mixed outcome fields

The gates already agreed in this thread validate each row. These columns validate the set of rows — which is what the bootstrap actually resamples.

None of this is theoretical caution. Every section above is a conclusion we nearly shipped wrong: the warm-arm comparison, the drifted sweep, the averaged mixture, the buried variance win, the mixed metric. The gates caught none of them, because the gates audit records and these live between records. When your numbers land, this is the layer I'd most like to compare notes on.