Three ways a token meter lies (and what it took to make ours honest)

A follow-up to "What actually counts as evidence for a token-saving claim." Link; https://forum.cursor.com/t/what-should-count-as-evidence-for-a-token-saving-agent-tool/166114/11 The discussion that post started kept circling two questions: how do you separate measured usage from estimated usage, and does your per-request accounting split cached input from fresh input — because without that split, you can't tell a real reduction from a caching artifact.

We build a local-first AI code editor, and we've been burned by all three versions of this. Our token meter has lied to us three different ways, each one documented in our changelog with the fix and a regression test. Here's the field report.

Lie #1: the meter that read zero

For months, our per-conversation token totals and the usage-history grid read empty. Not low — empty. When we finally inspected the on-disk database, the usage table had zero rows despite months of multi-message chats.

The persistence layer was fine. The bug was upstream: our gateway — the proxy every default-model request routed through — never forwarded the provider's usage field. The client's accumulator hit if (!usage) return and silently no-op'd. One dropped field killed three surfaces at once: the live meter, the per-chat totals, and the history buckets.

Here's why this matters for anyone benchmarking a "token-saving" tool: during that window, any measurement we ran would have shown spectacular savings. A broken meter is indistinguishable from a perfectly efficient tool. The fix was structural — normalize usage once at the gateway boundary into a single canonical shape ({in, out, cache_read, cache_creation}), covering OpenAI's prompt_tokens/completion_tokens shape, streaming responses (which require explicitly asking for the final usage chunk), and even local Ollama models (whose prompt_eval_count/eval_count were invisible too).

The durable lesson is a gate: a run with missing or zero usage is an invalid run, not a cheap one. The wiring check people rightly demand for the execution path ("does the measured config actually reach the shipped path?") applies equally to the measurement path. Both can silently disconnect.

Lie #2: the estimate that was 7× the bill

Our in-app cost readout said a session cost ~$25. The provider's own console said the same session billed ~$3.40.

The estimator priced input, output, and cache tokens at one blended rate. In reality, output tokens cost roughly 5× input, and a cache read costs roughly 10% of input. With a cache-heavy workload, a single blended rate isn't a rounding error — it's a 7× distortion, delivered with full confidence and no error message.

The fix was per-model, per-token-type pricing with provider-specific cache math. But the part worth stealing is the verification: we locked in a test that replays an actual observed session — 615,511 input, 19,161 output, 140,992 cache-read, 474,461 cache-write tokens — and asserts it prices to the number the provider console showed. An estimate that has never been reconciled against a real invoice isn't "estimated," it's unverified. Label it accordingly, and reconcile at least once.

Lie #3: the bill that 6×'d with zero behavior change

A routine 4-call agent session read only 4,329 of 98,741 input tokens from cache — a 4% hit rate on a workload that should sit near 95% — and re-wrote 94,404 tokens of cache. $0.59 of that session's $0.60 was cache writes.

No model changed. No tool changed. No prompt changed. What changed was an internal history-trimming detail: at the turn cap, the message array got trimmed to exactly the cap on every persist, which shifted the byte prefix every iteration and invalidated the cache on every single call. The fix (trim with hysteresis, so persists stay append-only between trims) took the at-cap rewrite fraction from 1.00 to 0.22 and the shared prefix from ~20% to ~87% — an honest ~6× cost reduction (~$0.60 → ~$0.10 for that session shape) from a change no benchmark of "the tool" would attribute correctly.

This cuts both ways. A caching regression can bury a real tool improvement, and a caching improvement can masquerade as one. If your per-run record doesn't split cache-read from cache-write from fresh input, these are all just "input tokens went up/down" and you will tell yourself the wrong story. We now surface a cache-hit percentage next to the token count in the app, plus a tripwire that warns when a conversation's read rate drops below 30% — because this failure mode returns quietly.

The split exists at the source — but the semantics differ

The good news: major providers already report the cached/fresh split in every API response. The trap: they disagree about what "input tokens" means.

• Anthropic reports three separate fields — input_tokens, cache_read_input_tokens, cache_creation_input_tokens — and input_tokens is the uncached remainder only. Total prompt size is the sum of all three.
• OpenAI reports prompt_tokens including cached tokens, with the cached portion broken out under prompt_tokens_details.cached_tokens.

Sum "input tokens" naively across providers and you're comparing different quantities. The price asymmetry is why the split matters economically: cache reads bill at ~10% of input rate, cache writes at a premium over input rate. Two runs with identical "total input" can differ several-fold in cost. Normalize once, at one boundary, into one canonical shape — and record the raw provider fields alongside it so you can audit the normalization later.

The record we keep now

Every run that feeds a claim gets one row:

• provider-reported (measured) usage: fresh input, cache-read, cache-write, output — never client-side estimates in the same column
• rounds and retries
• the verified task outcome, in the same row as the usage

That last one is non-negotiable, and our sharpest evidence for it is the inverse case: our fine-tune's real win showed up as +22 points of apply-rate — the outcome column — while its token delta was noise-level (~1%), a number we deliberately don't claim. Track usage alone and you'd have called the whole thing a wash. Cost and quality are one experiment, not two.

The gate list, updated

If you're evaluating a token-saving tool, in order:

1. Wiring gate — the measured configuration reaches the shipped execution path.
2. Meter gate — usage is provider-reported, present, and non-zero; missing usage invalidates the run.
3. Reconciliation — any estimated cost has been checked against the provider's console at least once, and the check is locked in a test.
4. Split-aware denominator — cache-read, cache-write, and fresh input recorded separately, normalized across providers.
5. Paired outcome — usage and verified success rate in the same record, always reported together.

Then, and only then, do the statistics from the first post — n≥5, bootstrap CIs, tier stratification — start meaning something. The meter is part of the experiment.