Tyler Garrett

This one is for the re-run @yohjisakamoto described: raw provider usage objects, normalized fresh/cache-read/cache-write splits, measured vs estimated buckets, n≥5 with bootstrap CIs and tier stratification, wiring and meter checks as preconditions. That list is right, and it's more rigor than any public comparison of these tools has had. But every gate in it validates a single run's record. Nothing in it validates the sample — and the statistics you're about to compute assume something about the sample that agentic benchmark runs quietly violate.

A bootstrap CI treats your runs as independent draws from one distribution. In this domain they aren't, in at least five specific ways. Each of these cost us a wrong conclusion (or nearly did) before we found it, and none of them appear in any methodology writeup we've seen. Consider this the field guide we wish someone had handed us.

1. The cache couples consecutive runs

Provider prompt caches persist for minutes to an hour after a request. Run configuration A, then configuration B on the same task, and B inherits whatever prefix cache A just paid to write. Whichever arm runs second looks cheaper — regardless of the tool.

This is not a small term. In our own editor, cache state alone moved a session's bill ~6× with zero change to model, tool, or prompt (the cache-churn incident from the earlier post: 4% read rate where ~95% was expected, $0.59 of a $0.60 session in cache writes). When 96% of tokens are cached input, cache state is a bigger term than most of the tool effects anyone is hunting. A comparison that doesn't control it is measuring cache weather.

What we do about it:

• Interleave arms, never run them in blocks. A-A-A-A-B-B-B-B maximally confounds arm with cache state (and with time — see §2). Alternate or randomize.
• Decide a cold-start policy and record it. Either force cold (gaps longer than the cache TTL, or a distinct prefix per run) or give both arms the same warmup and drop run #1 from each.
• Keep fresh-input as its own column. With the read/write/fresh split recorded per run, you can compute the comparison both on total billed tokens and on fresh input alone. When those two disagree, cache coupling is the reason — and that disagreement is itself a finding.

2. A sweep is a time series, and the system drifts under it

An n≥5-per-cell, tier-stratified sweep takes days. Providers ship silent model updates on that timescale. Cursor ships harness updates on that timescale. Pricing tables drift on that timescale — our own cost readout was confidently 7× wrong for a while, and that was client-side drift; the provider-side kind you can't even read in a diff.

If arm A ran Monday and arm B ran Thursday, you compared two different worlds and labeled the difference "the tool."

What we do about it:

• Our measurement harness interleaves per step — each step is a complete model×variant comparison, so drift lands on both arms equally instead of on one.
• Record the model ID from the response, not the request. The request says what you asked for; the response says what actually served the run.
• Record harness and plugin versions in the run row. If either changes mid-sweep, the sweep has a seam — analyze the halves separately or start over.

3. Your sample is a mixture, and the mean of a mixture lies

Our noisiest model refused to produce a stable token number: a 6.4× range (4,091 to 26,358 tokens) on the same task, CV ~50%. The temptation is to call that randomness and average harder. It wasn't randomness. Reading the logged commands showed a distinct failure mode — the model used the terminal as its primary interface, spelunking the shell (cat, sed, find, grep, re-running the same build) instead of using its tools, and how deep it dug was the variance.

That means the sample wasn't one distribution. It was a mixture: clean runs and spelunking runs. And interventions act on the mixture, not the mean — the directive we shipped for this cut the mean −38% (n=12), but its real effect was killing the catastrophic tail: worst case dropped 47%, from 34,154 to 18,159 tokens.

What we do about it:

• Classify every run before averaging: completed clean / wandered / retried / failed. The classification comes from the logs, and it's cheap.
• Report the mixture shift next to the mean shift. A tool that converts wanderers into clean runs can leave the median untouched and still be the most valuable thing you tested. Averaged blindly, it looks like nothing.
• Work in log space (multiplicative costs, heavy tails) — the log-ratio bootstrap from earlier in this thread exists because one 26k-token run should not own your mean.

4. Variance is a result, not an error bar

The most robust finding in our whole efficiency investigation wasn't a mean. On one reasoning model, our intervention read −32% mean tokens — but the number that replicated cleanly was the variance collapse: CV 75% → 22%. Runs went from swinging 6× to landing in a tight band. (An earlier n=2 probe of the same effect read −63%; the direction was robust, the magnitude was n-dependent. Treat your own early magnitudes the same way.)

Two tools with identical means and 3× different spread are not equivalent products. The tight one you can budget; the wide one hands you a surprise bill some fraction of the time. If the re-run reports only means and CIs, this dimension is invisible.

Report per-arm CV (or IQR) as a first-class result. A variance change with a flat mean is worth publishing on its own.

5. The verifier is part of the instrument

Billed-usage-per-verified-success puts a verifier inside the measurement loop, and the verifier has its own failure modes. Two of ours:

• Metric mixing. Our headline eval score climbed impressively — and decomposition showed it climbed because apply-rate climbed (did the edit apply at all), not correctness. One score was silently blending "produced output in the right shape" with "produced the right output." De-mix before you trust a denominator: applied? passed real tests? honest about what it did?
• The too-good score. A 100% result turned out to be train/eval leakage — the system had effectively seen the answers. Your version of this is subtler: repeated runs of the same task pool, with caching and retrieval in the loop, can "learn" the benchmark across repetitions. A score that improves across repeats of an unchanged configuration is a leak detector firing.

And if verification is an LLM judge: pin its version, record it per run, and accept that its variance is now inside your CI. Given the coupling point from earlier in this thread (compression moves usage and pass rate), a drifting verifier makes any usage delta unfalsifiable — you can no longer say whether the bill moved or the bar did.

What this adds to the run row

The record schema this thread converged on, plus the sample-level columns:

• interleave position / arm order index, and timestamps
• cold/warm cache flag (and the fresh-vs-total comparison both ways)
• model ID as reported by the response, harness + plugin versions
• run classification (clean / wandered / retried / failed), from logs
• verifier ID + version, with de-mixed outcome fields

The gates already agreed in this thread validate each row. These columns validate the set of rows — which is what the bootstrap actually resamples.

None of this is theoretical caution. Every section above is a conclusion we nearly shipped wrong: the warm-arm comparison, the drifted sweep, the averaged mixture, the buried variance win, the mixed metric. The gates caught none of them, because the gates audit records and these live between records. When your numbers land, this is the layer I'd most like to compare notes on.

A quick note, whenever you use any contact page, they might be able to track you. Today, we are tracking the IP address of any spammy signals, and then we begin communicating with that hosting company to understand more. We do this because it's exciting and fun. It's the game we must play online, and we accept your request to play. See you on the digital battle fields.

The reason j7 was created to be a honey pot for attention, and get the attention off of the priority solutions. This is saving a ton of time/money having to support individual websites under attack. Welcome to 2026. Hackers have ai, and so do we.

A follow-up to "What actually counts as evidence for a token-saving claim." Link; https://forum.cursor.com/t/what-should-count-as-evidence-for-a-token-saving-agent-tool/166114/11 The discussion that post started kept circling two questions: how do you separate measured usage from estimated usage, and does your per-request accounting split cached input from fresh input — because without that split, you can't tell a real reduction from a caching artifact.

We build a local-first AI code editor, and we've been burned by all three versions of this. Our token meter has lied to us three different ways, each one documented in our changelog with the fix and a regression test. Here's the field report.

Lie #1: the meter that read zero

For months, our per-conversation token totals and the usage-history grid read empty. Not low — empty. When we finally inspected the on-disk database, the usage table had zero rows despite months of multi-message chats.

The persistence layer was fine. The bug was upstream: our gateway — the proxy every default-model request routed through — never forwarded the provider's usage field. The client's accumulator hit if (!usage) return and silently no-op'd. One dropped field killed three surfaces at once: the live meter, the per-chat totals, and the history buckets.

Here's why this matters for anyone benchmarking a "token-saving" tool: during that window, any measurement we ran would have shown spectacular savings. A broken meter is indistinguishable from a perfectly efficient tool. The fix was structural — normalize usage once at the gateway boundary into a single canonical shape ({in, out, cache_read, cache_creation}), covering OpenAI's prompt_tokens/completion_tokens shape, streaming responses (which require explicitly asking for the final usage chunk), and even local Ollama models (whose prompt_eval_count/eval_count were invisible too).

The durable lesson is a gate: a run with missing or zero usage is an invalid run, not a cheap one. The wiring check people rightly demand for the execution path ("does the measured config actually reach the shipped path?") applies equally to the measurement path. Both can silently disconnect.

Lie #2: the estimate that was 7× the bill

Our in-app cost readout said a session cost ~$25. The provider's own console said the same session billed ~$3.40.

The estimator priced input, output, and cache tokens at one blended rate. In reality, output tokens cost roughly 5× input, and a cache read costs roughly 10% of input. With a cache-heavy workload, a single blended rate isn't a rounding error — it's a 7× distortion, delivered with full confidence and no error message.

The fix was per-model, per-token-type pricing with provider-specific cache math. But the part worth stealing is the verification: we locked in a test that replays an actual observed session — 615,511 input, 19,161 output, 140,992 cache-read, 474,461 cache-write tokens — and asserts it prices to the number the provider console showed. An estimate that has never been reconciled against a real invoice isn't "estimated," it's unverified. Label it accordingly, and reconcile at least once.

Lie #3: the bill that 6×'d with zero behavior change

A routine 4-call agent session read only 4,329 of 98,741 input tokens from cache — a 4% hit rate on a workload that should sit near 95% — and re-wrote 94,404 tokens of cache. $0.59 of that session's $0.60 was cache writes.

No model changed. No tool changed. No prompt changed. What changed was an internal history-trimming detail: at the turn cap, the message array got trimmed to exactly the cap on every persist, which shifted the byte prefix every iteration and invalidated the cache on every single call. The fix (trim with hysteresis, so persists stay append-only between trims) took the at-cap rewrite fraction from 1.00 to 0.22 and the shared prefix from ~20% to ~87% — an honest ~6× cost reduction (~$0.60 → ~$0.10 for that session shape) from a change no benchmark of "the tool" would attribute correctly.

This cuts both ways. A caching regression can bury a real tool improvement, and a caching improvement can masquerade as one. If your per-run record doesn't split cache-read from cache-write from fresh input, these are all just "input tokens went up/down" and you will tell yourself the wrong story. We now surface a cache-hit percentage next to the token count in the app, plus a tripwire that warns when a conversation's read rate drops below 30% — because this failure mode returns quietly.

The split exists at the source — but the semantics differ

The good news: major providers already report the cached/fresh split in every API response. The trap: they disagree about what "input tokens" means.

• Anthropic reports three separate fields — input_tokens, cache_read_input_tokens, cache_creation_input_tokens — and input_tokens is the uncached remainder only. Total prompt size is the sum of all three.
• OpenAI reports prompt_tokens including cached tokens, with the cached portion broken out under prompt_tokens_details.cached_tokens.

Sum "input tokens" naively across providers and you're comparing different quantities. The price asymmetry is why the split matters economically: cache reads bill at ~10% of input rate, cache writes at a premium over input rate. Two runs with identical "total input" can differ several-fold in cost. Normalize once, at one boundary, into one canonical shape — and record the raw provider fields alongside it so you can audit the normalization later.

The record we keep now

Every run that feeds a claim gets one row:

• provider-reported (measured) usage: fresh input, cache-read, cache-write, output — never client-side estimates in the same column
• rounds and retries
• the verified task outcome, in the same row as the usage

That last one is non-negotiable, and our sharpest evidence for it is the inverse case: our fine-tune's real win showed up as +22 points of apply-rate — the outcome column — while its token delta was noise-level (~1%), a number we deliberately don't claim. Track usage alone and you'd have called the whole thing a wash. Cost and quality are one experiment, not two.

The gate list, updated

If you're evaluating a token-saving tool, in order:

1. Wiring gate — the measured configuration reaches the shipped execution path.
2. Meter gate — usage is provider-reported, present, and non-zero; missing usage invalidates the run.
3. Reconciliation — any estimated cost has been checked against the provider's console at least once, and the check is locked in a test.
4. Split-aware denominator — cache-read, cache-write, and fresh input recorded separately, normalized across providers.
5. Paired outcome — usage and verified success rate in the same record, always reported together.

Then, and only then, do the statistics from the first post — n≥5, bootstrap CIs, tier stratification — start meaning something. The meter is part of the experiment.

The AI race has largely been framed as a competition around speed, benchmarks, and pricing. Every few weeks another model appears claiming to outperform the last one.

But Kimi K3 deserves scrutiny for reasons beyond its benchmark scores. (to my knowledge no one has access to the benchmark scores, or a harness to evaluate that won't lie to them)

There are two separate conversations happening around Kimi, and they're often conflated:

1. Did Moonshot AI build Kimi by improperly extracting knowledge from proprietary American models?
2. Should businesses trust the hosted service with confidential information?

These are different questions. Both matter.

Allegations of Model IP Theft

One of the most significant accusations surrounding Kimi K3 is that its capabilities may have been built through unauthorized model distillation.

Model distillation is a legitimate machine learning technique when performed with permission. The controversy arises when a company repeatedly queries a proprietary model at massive scale, captures its outputs, and uses those outputs to train a competing system without authorization.

According to public reporting and statements from U.S. officials, Moonshot AI allegedly generated millions of interactions with Anthropic's Claude using large numbers of accounts in an effort to recreate the model's behavior.

If accurate, this isn't simply "learning from public information."

It's effectively using another company's commercial product as the training dataset.

That distinction matters.

The "I'm Claude" Incident

Shortly after Kimi K3's release, users discovered prompts where the model identified itself as:

"I'm Claude, an AI assistant made by Anthropic."

Large language models hallucinate, and isolated responses alone are not definitive proof of copying.

However, identity confusion of this type has historically raised questions about how a model was trained or evaluated. When combined with broader allegations of unauthorized distillation, these examples naturally attracted additional attention.

On their own they prove very little.

In context, they become more interesting.

Government Attention

The issue has moved beyond internet speculation.

Senior U.S. officials have publicly discussed stronger responses to foreign AI companies accused of stealing American intellectual property, including the possibility of sanctions and export-related restrictions.

Whether any enforcement ultimately occurs remains to be seen, but the allegations are being treated seriously enough to become part of national technology policy discussions.

Your Intellectual Property Is a Different Conversation

Even if the model itself were completely legitimate, another question remains:

What happens to the information you submit?

If you're using the hosted version of Kimi through its website or application, your prompts are processed on infrastructure operated by the provider.

For organizations handling:

• proprietary software
• unreleased products
• customer information
• confidential legal documents
• internal business strategy

that should immediately trigger a data governance review.

Different jurisdictions have different legal frameworks governing government access to hosted data. Organizations operating under privacy regulations, contractual confidentiality obligations, or non-disclosure agreements should understand exactly where their information is processed before uploading sensitive material.

Convenience should not replace due diligence.

Narrative Control and Online Promotion

Another concern occasionally raised online is the unusually enthusiastic promotion surrounding Kimi.

Some users have alleged that coordinated social media accounts, automated posting, or bot networks are being used to amplify positive sentiment and suppress criticism.

No publicly available evidence has established that Moonshot AI operates or directs bot networks for this purpose. The claim remains speculative unless supported by verifiable evidence.

That said, coordinated influence campaigns do exist across social media, and artificial engagement has become increasingly common across many industries. Distinguishing genuine community enthusiasm from manufactured consensus requires careful analysis—not assumptions.

Open Weights Change the Equation

Moonshot AI has discussed releasing open model weights.

If organizations wish to evaluate the model, running it locally offers a significantly different risk profile than relying on a hosted service.

A locally deployed model allows organizations to:

• keep proprietary data inside their own infrastructure
• control logging and retention
• comply more easily with internal security policies
• reduce exposure of confidential prompts

For many engineering teams, local deployment is often preferable regardless of the model vendor.

The Bigger Issue

The AI industry increasingly depends on trust.

If companies expect their own models to be protected as intellectual property, that principle should apply consistently across the ecosystem.

Likewise, users deserve transparency about where their data goes and how it may be used.

Neither concern should be dismissed because a model performs well on benchmarks.

Performance and provenance are different questions.

Security and convenience are different tradeoffs.

As organizations adopt increasingly capable AI systems, understanding how a model was built and where your data goes may matter just as much as benchmark scores.

That's the post. Just feeling a lot better. Hustling around more. Not feeling a lot of pain or sensations anymore. Hopeful to be back training soon. However something to realize. I need to get, a lot, stronger...