Two Weeks Inside Yee: The Token-Honesty Audit, and What Came Out of It

We build Yee, a local-first AI coding editor. Most weeks we ship features. This last stretch turned into something a little different: we set out to audit how honestly the app measures its own token spend, and the audit kept finding new ways the meter had been lying to us — not maliciously, just quietly, the way instrumentation always does when nobody's checked it against reality in a while.

Here's what that turned into, and a few unrelated things that shipped alongside it.

The meter was wrong. Then it was wrong again. Then again.

It started with a simple complaint: the dollar estimate on a session read ~$25. The real Anthropic invoice for that same session was ~$3.4. Root cause — the estimator priced every token type (input, output, cache read, cache write) at one blended rate, when in reality output costs roughly 5x input and a cache read costs about 10% of input. Fixed by pricing each token type separately, per model, and locking the fix with a test that replays the exact real session and asserts it prices to $3.5 — not the old $25.

That fix made the meter trustworthy enough to notice the next problem: a real agent-loop session had read only 4,329 of 98,741 input tokens from cache — 4%, when message-prefix caching should have delivered something like 95%. Almost the entire session cost was cache writes, not reads. Tracing it down: at the max-turns cap, history trimming shifted the message array by exactly one slot on every turn, which broke the byte-identical prefix the cache depends on — every single call. The fix (a "floor" that trims down in one shot instead of shaving one message at a time) took the at-cap rewrite rate from 100% down to 22%, and the shared-prefix rate from ~20% up to ~87%. Measured honestly, in dollars: about $0.60 down to about $0.10 for the same work.

Then we found the meter wasn't just miscalibrated — for a whole class of sessions it was reading zero. The start-screen token-history grid was blank; per-conversation totals vanished on restart. The database had 0 rows in its usage-hours table despite months of real multi-message chats. The cause: the API gateway that the default model path routes through simply never forwarded a usage field in its responses, so the code that feeds the meter hit an early return and silently no-op'd — for the meter, the per-chat total, and the historical charts, all from one dropped field. Local-model (Ollama) spend had the same blind spot for the same reason. Fixed by normalizing and forwarding usage at the gateway boundary, for both backends, on both streaming and non-streaming paths.

Three separate, real bugs, found by comparing what the app claimed against what actually happened — not by design review, but by checking. That's the pattern we're trying to hold ourselves to: an instrument only earns trust once someone's tried to catch it lying and failed.

Alongside the audit: real caching, a real circuit breaker, and a local-model summarizer that costs nothing

The same two weeks shipped the actual token-saving mechanisms the meter now measures honestly:

• Message-prefix caching went live — every agent-loop call now cache-marks the growing conversation, not just the system prompt, so a live-probed follow-up call read 99.7% of its input from cache instead of paying full price to resend the whole transcript.
• A pinned task, an unpinned middle. History eviction used to drop the original task first, since it was the oldest message — exactly the thing a long-running loop can't afford to forget. Now the first task+context pair is pinned for the life of the conversation, and only the middle evicts.
• A real budget circuit breaker. Nothing used to stand between a runaway loop and an unbounded bill. Now a soft warning lands at 150k tokens per turn, and a hard pause at 300k — with a "keep going" button that requires an explicit choice to spend another 300k, and (as of this stretch) an actual stop option, since the pause used to only offer "keep going" and the send button would silently do nothing mid-run if you tried to interrupt it.
• Context compaction that costs zero API tokens. When history has to be dropped, instead of just deleting it, Yee now asks the locally-forged model — running on your own hardware — to summarize the span being evicted. It falls back silently to plain eviction if the local model is slow or unavailable, so it never adds latency risk to a paid call.
• Native tool calling (Anthropic, opt-in via flag). The text-based tool protocol Yee uses everywhere else means every formatting slip costs a full extra round-trip to self-correct. Native mode lets the API carry tool calls directly, which removes that failure mode and about 600 tokens of format-teaching from the system prompt — currently opt-in while it proves itself.

Elsewhere in the app

Not everything this stretch was about tokens:

• A live system map. An interactive, hover-to-trace diagram of the whole editor — 38 nodes, 56 connections, built from an actual sweep of the code rather than hand-drawn, with each node honestly marked shipped, partial, or untested. If a feature isn't wired up yet, the map says so instead of implying otherwise.
• The training dashboard, embedded. The existing Forge dashboard — where edits you approve become local training data — now opens inside the editor itself as a third launch option, instead of living in a separate app you'd have to know to go find.
• Real Lighthouse scores in the chat header. Four live performance/accessibility/best-practices/SEO scores for whatever you're previewing, with a queue that lets you fix failing audits one focused turn at a time instead of dumping all nine problems into a single overwhelming prompt.

The takeaway

None of this is a "we're 50% cheaper now" headline, and we're deliberately not writing it as one — the honest number is that fixing three measurement bugs and one real caching bug got a specific session from about $0.60 to about $0.10, which is a real and meaningful improvement, not a universal multiplier. The more durable point is the process: build the thing, then go looking for the ways it's lying to you, and don't stop at the first bug you find.