Tyler Garrett

A follow-up to "What actually counts as evidence for a token-saving claim." Link; https://forum.cursor.com/t/what-should-count-as-evidence-for-a-token-saving-agent-tool/166114/11 The discussion that post started kept circling two questions: how do you separate measured usage from estimated usage, and does your per-request accounting split cached input from fresh input — because without that split, you can't tell a real reduction from a caching artifact.

We build a local-first AI code editor, and we've been burned by all three versions of this. Our token meter has lied to us three different ways, each one documented in our changelog with the fix and a regression test. Here's the field report.

Lie #1: the meter that read zero

For months, our per-conversation token totals and the usage-history grid read empty. Not low — empty. When we finally inspected the on-disk database, the usage table had zero rows despite months of multi-message chats.

The persistence layer was fine. The bug was upstream: our gateway — the proxy every default-model request routed through — never forwarded the provider's usage field. The client's accumulator hit if (!usage) return and silently no-op'd. One dropped field killed three surfaces at once: the live meter, the per-chat totals, and the history buckets.

Here's why this matters for anyone benchmarking a "token-saving" tool: during that window, any measurement we ran would have shown spectacular savings. A broken meter is indistinguishable from a perfectly efficient tool. The fix was structural — normalize usage once at the gateway boundary into a single canonical shape ({in, out, cache_read, cache_creation}), covering OpenAI's prompt_tokens/completion_tokens shape, streaming responses (which require explicitly asking for the final usage chunk), and even local Ollama models (whose prompt_eval_count/eval_count were invisible too).

The durable lesson is a gate: a run with missing or zero usage is an invalid run, not a cheap one. The wiring check people rightly demand for the execution path ("does the measured config actually reach the shipped path?") applies equally to the measurement path. Both can silently disconnect.

Lie #2: the estimate that was 7× the bill

Our in-app cost readout said a session cost ~$25. The provider's own console said the same session billed ~$3.40.

The estimator priced input, output, and cache tokens at one blended rate. In reality, output tokens cost roughly 5× input, and a cache read costs roughly 10% of input. With a cache-heavy workload, a single blended rate isn't a rounding error — it's a 7× distortion, delivered with full confidence and no error message.

The fix was per-model, per-token-type pricing with provider-specific cache math. But the part worth stealing is the verification: we locked in a test that replays an actual observed session — 615,511 input, 19,161 output, 140,992 cache-read, 474,461 cache-write tokens — and asserts it prices to the number the provider console showed. An estimate that has never been reconciled against a real invoice isn't "estimated," it's unverified. Label it accordingly, and reconcile at least once.

Lie #3: the bill that 6×'d with zero behavior change

A routine 4-call agent session read only 4,329 of 98,741 input tokens from cache — a 4% hit rate on a workload that should sit near 95% — and re-wrote 94,404 tokens of cache. $0.59 of that session's $0.60 was cache writes.

No model changed. No tool changed. No prompt changed. What changed was an internal history-trimming detail: at the turn cap, the message array got trimmed to exactly the cap on every persist, which shifted the byte prefix every iteration and invalidated the cache on every single call. The fix (trim with hysteresis, so persists stay append-only between trims) took the at-cap rewrite fraction from 1.00 to 0.22 and the shared prefix from ~20% to ~87% — an honest ~6× cost reduction (~$0.60 → ~$0.10 for that session shape) from a change no benchmark of "the tool" would attribute correctly.

This cuts both ways. A caching regression can bury a real tool improvement, and a caching improvement can masquerade as one. If your per-run record doesn't split cache-read from cache-write from fresh input, these are all just "input tokens went up/down" and you will tell yourself the wrong story. We now surface a cache-hit percentage next to the token count in the app, plus a tripwire that warns when a conversation's read rate drops below 30% — because this failure mode returns quietly.

The split exists at the source — but the semantics differ

The good news: major providers already report the cached/fresh split in every API response. The trap: they disagree about what "input tokens" means.

• Anthropic reports three separate fields — input_tokens, cache_read_input_tokens, cache_creation_input_tokens — and input_tokens is the uncached remainder only. Total prompt size is the sum of all three.
• OpenAI reports prompt_tokens including cached tokens, with the cached portion broken out under prompt_tokens_details.cached_tokens.

Sum "input tokens" naively across providers and you're comparing different quantities. The price asymmetry is why the split matters economically: cache reads bill at ~10% of input rate, cache writes at a premium over input rate. Two runs with identical "total input" can differ several-fold in cost. Normalize once, at one boundary, into one canonical shape — and record the raw provider fields alongside it so you can audit the normalization later.

The record we keep now

Every run that feeds a claim gets one row:

• provider-reported (measured) usage: fresh input, cache-read, cache-write, output — never client-side estimates in the same column
• rounds and retries
• the verified task outcome, in the same row as the usage

That last one is non-negotiable, and our sharpest evidence for it is the inverse case: our fine-tune's real win showed up as +22 points of apply-rate — the outcome column — while its token delta was noise-level (~1%), a number we deliberately don't claim. Track usage alone and you'd have called the whole thing a wash. Cost and quality are one experiment, not two.

The gate list, updated

If you're evaluating a token-saving tool, in order:

1. Wiring gate — the measured configuration reaches the shipped execution path.
2. Meter gate — usage is provider-reported, present, and non-zero; missing usage invalidates the run.
3. Reconciliation — any estimated cost has been checked against the provider's console at least once, and the check is locked in a test.
4. Split-aware denominator — cache-read, cache-write, and fresh input recorded separately, normalized across providers.
5. Paired outcome — usage and verified success rate in the same record, always reported together.

Then, and only then, do the statistics from the first post — n≥5, bootstrap CIs, tier stratification — start meaning something. The meter is part of the experiment.

The AI race has largely been framed as a competition around speed, benchmarks, and pricing. Every few weeks another model appears claiming to outperform the last one.

But Kimi K3 deserves scrutiny for reasons beyond its benchmark scores. (to my knowledge no one has access to the benchmark scores, or a harness to evaluate that won't lie to them)

There are two separate conversations happening around Kimi, and they're often conflated:

1. Did Moonshot AI build Kimi by improperly extracting knowledge from proprietary American models?
2. Should businesses trust the hosted service with confidential information?

These are different questions. Both matter.

Allegations of Model IP Theft

One of the most significant accusations surrounding Kimi K3 is that its capabilities may have been built through unauthorized model distillation.

Model distillation is a legitimate machine learning technique when performed with permission. The controversy arises when a company repeatedly queries a proprietary model at massive scale, captures its outputs, and uses those outputs to train a competing system without authorization.

According to public reporting and statements from U.S. officials, Moonshot AI allegedly generated millions of interactions with Anthropic's Claude using large numbers of accounts in an effort to recreate the model's behavior.

If accurate, this isn't simply "learning from public information."

It's effectively using another company's commercial product as the training dataset.

That distinction matters.

The "I'm Claude" Incident

Shortly after Kimi K3's release, users discovered prompts where the model identified itself as:

"I'm Claude, an AI assistant made by Anthropic."

Large language models hallucinate, and isolated responses alone are not definitive proof of copying.

However, identity confusion of this type has historically raised questions about how a model was trained or evaluated. When combined with broader allegations of unauthorized distillation, these examples naturally attracted additional attention.

On their own they prove very little.

In context, they become more interesting.

Government Attention

The issue has moved beyond internet speculation.

Senior U.S. officials have publicly discussed stronger responses to foreign AI companies accused of stealing American intellectual property, including the possibility of sanctions and export-related restrictions.

Whether any enforcement ultimately occurs remains to be seen, but the allegations are being treated seriously enough to become part of national technology policy discussions.

Your Intellectual Property Is a Different Conversation

Even if the model itself were completely legitimate, another question remains:

What happens to the information you submit?

If you're using the hosted version of Kimi through its website or application, your prompts are processed on infrastructure operated by the provider.

For organizations handling:

• proprietary software
• unreleased products
• customer information
• confidential legal documents
• internal business strategy

that should immediately trigger a data governance review.

Different jurisdictions have different legal frameworks governing government access to hosted data. Organizations operating under privacy regulations, contractual confidentiality obligations, or non-disclosure agreements should understand exactly where their information is processed before uploading sensitive material.

Convenience should not replace due diligence.

Narrative Control and Online Promotion

Another concern occasionally raised online is the unusually enthusiastic promotion surrounding Kimi.

Some users have alleged that coordinated social media accounts, automated posting, or bot networks are being used to amplify positive sentiment and suppress criticism.

No publicly available evidence has established that Moonshot AI operates or directs bot networks for this purpose. The claim remains speculative unless supported by verifiable evidence.

That said, coordinated influence campaigns do exist across social media, and artificial engagement has become increasingly common across many industries. Distinguishing genuine community enthusiasm from manufactured consensus requires careful analysis—not assumptions.

Open Weights Change the Equation

Moonshot AI has discussed releasing open model weights.

If organizations wish to evaluate the model, running it locally offers a significantly different risk profile than relying on a hosted service.

A locally deployed model allows organizations to:

• keep proprietary data inside their own infrastructure
• control logging and retention
• comply more easily with internal security policies
• reduce exposure of confidential prompts

For many engineering teams, local deployment is often preferable regardless of the model vendor.

The Bigger Issue

The AI industry increasingly depends on trust.

If companies expect their own models to be protected as intellectual property, that principle should apply consistently across the ecosystem.

Likewise, users deserve transparency about where their data goes and how it may be used.

Neither concern should be dismissed because a model performs well on benchmarks.

Performance and provenance are different questions.

Security and convenience are different tradeoffs.

As organizations adopt increasingly capable AI systems, understanding how a model was built and where your data goes may matter just as much as benchmark scores.

That's the post. Just feeling a lot better. Hustling around more. Not feeling a lot of pain or sensations anymore. Hopeful to be back training soon. However something to realize. I need to get, a lot, stronger...

Most AI coding tools measure themselves by what the model can do. We spent the beta measuring what it shouldn't have to do — every token the agent burns on something the UI could've just done directly is a token wasted. That discipline is the actual product.

The under-the-hood stuff

Every agent run is capped at 10 steps, on purpose

The loop doesn't wander. It gets 10 iterations to read, edit, and verify — then it stops and hands control back, rather than spiraling through redundant re-reads and "let me also check…" tangents. A single orchestrated step (read → edit → done) gets an even tighter cap. Slower isn't the failure mode we're avoiding — runaway spend is.

A real token budget gate, not a vibes-based one

At 150k tokens spent on a single request, you get a heads-up. At 300k, the loop physically pauses and waits for you to say "keep going." No silent meter climbing in the background — the system stops itself and asks.

The preview button starts the app. The AI doesn't.

This is the one that actually matters: when you click preview, the editor spins up the real dev server directly — no LLM call involved. Every system prompt in the agent explicitly forbids the model from running npm run dev, vite, astro dev, or any serve command to "check its work," because the dev server is already running and hot-reloading. Asking the AI to start your app is asking it to spend tokens duplicating something a button already did for free. We removed that whole category of waste.

Cache-aware by construction

The conversation window is managed to keep a stable prefix instead of shifting on every turn — because a byte-for-byte stable prefix is what makes prompt caching actually work. Get this wrong and you silently re-pay for the same context on every single call. We measured this directly and tuned the eviction policy around it, not around convenience.

Why this is the beta, not a footnote

Anyone can wire an agent loop to a model and call it a coding assistant. The interesting engineering problem is making that loop cheap to run for hours — bounded steps, a real spend gate, and infrastructure (preview, hot-reload) that does its job without asking the model to do it for you. That's the ecosystem: not just an assistant that codes, but one that knows when not to think.

Try it

YeeYee is a local-first Electron app. Beta access details to follow.

Made by dev3lop.com.

We build Yee, a local-first AI coding editor. Most weeks we ship features. This last stretch turned into something a little different: we set out to audit how honestly the app measures its own token spend, and the audit kept finding new ways the meter had been lying to us — not maliciously, just quietly, the way instrumentation always does when nobody's checked it against reality in a while.

Here's what that turned into, and a few unrelated things that shipped alongside it.

The meter was wrong. Then it was wrong again. Then again.

It started with a simple complaint: the dollar estimate on a session read ~$25. The real Anthropic invoice for that same session was ~$3.4. Root cause — the estimator priced every token type (input, output, cache read, cache write) at one blended rate, when in reality output costs roughly 5x input and a cache read costs about 10% of input. Fixed by pricing each token type separately, per model, and locking the fix with a test that replays the exact real session and asserts it prices to $3.5 — not the old $25.

That fix made the meter trustworthy enough to notice the next problem: a real agent-loop session had read only 4,329 of 98,741 input tokens from cache — 4%, when message-prefix caching should have delivered something like 95%. Almost the entire session cost was cache writes, not reads. Tracing it down: at the max-turns cap, history trimming shifted the message array by exactly one slot on every turn, which broke the byte-identical prefix the cache depends on — every single call. The fix (a "floor" that trims down in one shot instead of shaving one message at a time) took the at-cap rewrite rate from 100% down to 22%, and the shared-prefix rate from ~20% up to ~87%. Measured honestly, in dollars: about $0.60 down to about $0.10 for the same work.

Then we found the meter wasn't just miscalibrated — for a whole class of sessions it was reading zero. The start-screen token-history grid was blank; per-conversation totals vanished on restart. The database had 0 rows in its usage-hours table despite months of real multi-message chats. The cause: the API gateway that the default model path routes through simply never forwarded a usage field in its responses, so the code that feeds the meter hit an early return and silently no-op'd — for the meter, the per-chat total, and the historical charts, all from one dropped field. Local-model (Ollama) spend had the same blind spot for the same reason. Fixed by normalizing and forwarding usage at the gateway boundary, for both backends, on both streaming and non-streaming paths.

Three separate, real bugs, found by comparing what the app claimed against what actually happened — not by design review, but by checking. That's the pattern we're trying to hold ourselves to: an instrument only earns trust once someone's tried to catch it lying and failed.

Alongside the audit: real caching, a real circuit breaker, and a local-model summarizer that costs nothing

The same two weeks shipped the actual token-saving mechanisms the meter now measures honestly:

• Message-prefix caching went live — every agent-loop call now cache-marks the growing conversation, not just the system prompt, so a live-probed follow-up call read 99.7% of its input from cache instead of paying full price to resend the whole transcript.
• A pinned task, an unpinned middle. History eviction used to drop the original task first, since it was the oldest message — exactly the thing a long-running loop can't afford to forget. Now the first task+context pair is pinned for the life of the conversation, and only the middle evicts.
• A real budget circuit breaker. Nothing used to stand between a runaway loop and an unbounded bill. Now a soft warning lands at 150k tokens per turn, and a hard pause at 300k — with a "keep going" button that requires an explicit choice to spend another 300k, and (as of this stretch) an actual stop option, since the pause used to only offer "keep going" and the send button would silently do nothing mid-run if you tried to interrupt it.
• Context compaction that costs zero API tokens. When history has to be dropped, instead of just deleting it, Yee now asks the locally-forged model — running on your own hardware — to summarize the span being evicted. It falls back silently to plain eviction if the local model is slow or unavailable, so it never adds latency risk to a paid call.
• Native tool calling (Anthropic, opt-in via flag). The text-based tool protocol Yee uses everywhere else means every formatting slip costs a full extra round-trip to self-correct. Native mode lets the API carry tool calls directly, which removes that failure mode and about 600 tokens of format-teaching from the system prompt — currently opt-in while it proves itself.

Elsewhere in the app

Not everything this stretch was about tokens:

• A live system map. An interactive, hover-to-trace diagram of the whole editor — 38 nodes, 56 connections, built from an actual sweep of the code rather than hand-drawn, with each node honestly marked shipped, partial, or untested. If a feature isn't wired up yet, the map says so instead of implying otherwise.
• The training dashboard, embedded. The existing Forge dashboard — where edits you approve become local training data — now opens inside the editor itself as a third launch option, instead of living in a separate app you'd have to know to go find.
• Real Lighthouse scores in the chat header. Four live performance/accessibility/best-practices/SEO scores for whatever you're previewing, with a queue that lets you fix failing audits one focused turn at a time instead of dumping all nine problems into a single overwhelming prompt.

The takeaway

None of this is a "we're 50% cheaper now" headline, and we're deliberately not writing it as one — the honest number is that fixing three measurement bugs and one real caching bug got a specific session from about $0.60 to about $0.10, which is a real and meaningful improvement, not a universal multiplier. The more durable point is the process: build the thing, then go looking for the ways it's lying to you, and don't stop at the first bug you find.