Tyler Garrett

A follow-up to Cline lessons learned + feature roadmap for Yee easy mode.

Last time I published a competitive study of Cline, eighteen lessons, and a ranked backlog: 26 features, broken into 81 steps, each scored for complexity, sequenced across five milestones. It ended with "research complete — roadmap awaiting a ranking pass."

We did the ranking pass. Then we built it.

Where it stands: 51 of 81 rows shipped, 111 of 188 complexity points. Every row through the same gate — a hermetic test guard, the full chain green, an adversarial review, and a commit that says what it did. The test suite is now 74 steps and 2,510 assertions and costs nothing to run: no model call, no network, so it runs on every change.

That's the boring part. The interesting part is that executing a roadmap teaches you things writing one cannot, and three of those things were uncomfortable enough to be worth a post.

L19 — The roadmap was wrong in one specific, dangerous place

Row CL-04 was "classified command allowlist." The idea: instead of approving npm test over and over, approve the class of command once and stop being asked. Cline does something like this. It's an obvious quality-of-life win. I ranked it P1 and specced it in four steps.

Before writing any code, we ran three independent threat models against the design and had them judged.

The verdict: a grant keyed on a command's class is standing code execution.

The reasoning is simple once someone says it out loud. A class name doesn't bind to behavior. npm test doesn't mean "run tests" — it means "run whatever scripts.test currently says," and the agent editing your code can edit that. Grant the class, and you've granted anything that file can be made to say.

The obvious fix — hash the config files and re-ask when they change — doesn't hold either. scripts.test = node scripts/test.js moves the payload one level deeper than the hash can see. Same for tests/conftest.py, node_modules/.bin/*, and .git/config textconv drivers.

Three holes got executed against candidate designs rather than argued about:

• find . -exec inside a class called "inspect"
• git branch -D main inside a class the interface labelled read-only
• npx <tool>, which fetches and runs a package from a mutable registry

And the counterweight that killed the remaining case for it: the strict shell grammar you'd need to make classes safe refuses npx vitest run — this repo's own idiom. So the prompt-reduction payoff was small and the new attack surface was large.

What shipped instead: grants key on the exact command bytes you read in the modal, scoped to one project, surviving restarts. That inherits the session allowlist's safety property by construction — the granted string is the string that runs — while still killing the re-prompt-every-reload loop that trains people to click through without reading.

The lesson isn't "we caught a bug." It's that a roadmap row is a hypothesis, and some of them are wrong in ways only the build reveals. The row now carries a deviation note explaining what shipped instead and why, sitting next to the original spec. A backlog that only records intentions is a worse document than one that records collisions with reality.

L20 — A test suite can be green while the product lies

Here's a dialog Yee shipped:

No standing approval for this one — it uses "pnpm @dev3lop/site", which can change more than it reports. [ Deny ] [ Always allow this command ] [ Always allow here ] [ Run once ]

Read it twice. It says there is no standing approval available, directly beside an enabled button offering exactly that.

Every logic test passed. They were all correct. The button's condition was derived from one property (destructive) while the explanation was derived from another (grantable), and nothing in the suite had ever rendered the two together. A unit test can verify each half and never notice the pair contradicts itself, because the contradiction only exists on screen.

There was a second bug in the same screenshot: pnpm --filter @dev3lop/site build was refused because --filter consumes the next token, and its value was being read as the subcommand. Hence the nonsense message naming a package as if it were a command.

So we built a click-through harness. bench/app-drive.sh boots the real app hidden and evaluates scripts inside the real renderer — importing the shipped modules, opening the real modal, clicking the actual buttons, and asserting what a user would see. Zero tokens, and it runs in the chain like everything else.

87 clicks across five drivers now. The push modal, the approval modal, the endpoint settings, the secrets boundary through real IPC, the chat pills.

It earned itself on the first run by catching an unrelated regression: a refactor had collapsed rm -rf ./build into a generic "deletes files," losing the recursive-tree warning the modal used to give. No logic test would ever flag that — the string was still a valid reason.

If your interface makes claims, something has to read them.

L21 — Three of my own tests proved nothing, and mutation testing is how I know

This is the part I'd rather not write.

The discipline we adopted early: after writing a guard, seed the exact fault it claims to catch and confirm it goes red by name. Not "the suite fails" — that specific assertion, with its specific message.

It caught my own work three separate times.

The vacuous fixture. I wrote an assertion that the push modal names the branch it actually cuts from — the exact bug that had shipped and burned us. My test fixture set startBranch and defaultBranch both to main. So the assertion matched "off main" whether the code named the cut base or the branch you happened to be standing on. I reinstated the original bug and the test stayed green. It now reads the summary line with two different names.

The substring pin. I added a check that capturing the preview appends rather than replaces. It asserted a push(...) call existed in the source. I then seeded a fault that cleared the previous captures immediately before pushing — the substring still matched. Green. It reads the whole function body now.

The crash that looked like a pass. A seeded fault made a test crash mid-run. It printed no summary line, so grep FAILED found nothing and I read it as passing. Content assertions now short-circuit on error results so a mutation dies as a named red, not a silent stack trace.

Every one of those tests looked right. Two of them I'd have shipped and cited as proof.

A test that cannot fail is worse than no test, because it is counted as evidence.

The related habit that comes with it: vacuous-pass armor. Every guard sets process.exitCode = 1 and only a completed summary line flips it, plus a watchdog. Otherwise an async stall drains the event loop and exits 0 — reading as green. Suites also pin their own assertion count, so quietly losing a case is a failure rather than a smaller number nobody reads.

L22 — A number computed over lossy data is worse than no number

This one turned up three separate times in three unrelated features, which is how you know it's a real pattern and not an anecdote.

Phantom pricing. We added support for any OpenAI-compatible endpoint — OpenRouter, Groq, Together, LM Studio. The price resolver fell through to a default of $3/$15 per million tokens for anything it didn't recognise. Which meant a call to your own local server, costing nothing, would have been billed at Claude-tier rates into the spend meter, the receipts, and the generated PR body. We cannot know what a third party charges. So it's your number or an explicit unknown. Never a guess.

Silent truncation. The per-chat activity trace was capped at 400 entries with slice(-400) — which silently discards the earliest rows. Fine for a scrolling log. Fatal the moment you build an apply-success meter on it, because the rate gets computed over a biased tail and states a confidence it hasn't earned. The cap now leaves a marker recording what it dropped, and any statistic built on it reports itself as partial.

Discarded failures. Edits were only captured for training when they succeeded — the call site literally gated on success. Half the signal was going in the bin, and it was the more useful half: the instruction, the file, and the search block the model invented that matched nothing. Now failures are captured too, labelled negative at the schema level — because the row builder has a fixed field list, and a label set by the caller would have been silently dropped, putting a wrong edit into the training warehouse looking exactly like a win.

Same rule underneath all three: when you don't know, say so. An honest gap beats a confident fabrication, and it is much cheaper to fix later.

L23 — Stop treating the backlog as a queue

Halfway through, the sequencing stopped making sense.

M1 and M2 earned their place. They fixed spots where Yee was genuinely worse than table stakes — commands silently denied under concurrency, loops dead-ending with no way to continue, repo rules ignored, no way to restore a single turn. That work needed doing regardless of what Cline shipped.

Past that line, running the remaining rows in order converts into copying a better-funded team on their own axis. That's a race you lose by construction: you're always behind, by definition, and every hour spent matching them is an hour not spent on the things they can't answer.

So the grid became a menu. What gets built next is picked on one question: does this fix something that harms a real user, or widen something a competitor cannot answer?

Five rows survived that test, and they were not the next five in rank order:

Roughly the same complexity budget as one milestone. Far more value.

L24 — We are not building MCP, on purpose

Every agentic editor ships MCP. We held it, and I want to be precise about why, because "we didn't get to it" and "we decided not to" are different statements and only one of them is honest.

The case for is real, and includes a correction to my own reasoning. I argued for the compat-provider row because "one integration opens OpenRouter, Groq, LM Studio." MCP is the same shape at larger scale — one integration inherits hundreds of tool servers. I used that argument in one direction and not the other, and the difference is cost, not principle.

What's genuinely different is a cost specific to us. Yee's measured advantage is that its ecosystem — preview-awareness, redundant-command interception, a directness directive — makes frontier models spend fewer and steadier tokens for the same result. We have numbers on that. The standard MCP integration mirrors every connected server's tools into the turn-1 vocabulary and freezes it for cache stability, which means every chat pays context for every tool of every connected server, used or not. Connect three servers with a dozen tools each and you have quietly widened the exact thing you measured as your edge.

Add the rest: it's a permanent maintenance surface, and an MCP server is arbitrary code with tool access — running outside the secrets boundary we just spent a week building.

Nobody has ever chosen a code editor because it had MCP. They choose one because it does the thing they came for without lying to them.

It comes off hold the day a real user asks for a specific server by name — and when it does, the shape won't be "MCP support." It'll be the two or three servers that earn their context, enabled per-chat so a conversation that doesn't need a database never pays for one.

The backlog records this as held (⏸), deliberately distinct from not started (☐). A reader has to be able to tell an unmade decision from a made one.

L25 — Documentation rots in a specific, corrosive way

Catching the READMEs up after three weeks, I found the roadmap section listing three things as "later" that had already shipped: inline edit, @file mentions, signed macOS builds.

That's worse than merely out of date. There's an "honest list" in that README — the things Yee genuinely cannot do — and it's the most valuable section in the file. If a reader spots one item they know shipped, they stop trusting the honest list too. Stale optimism poisons stated limitations.

The honest list gained three entries in the same pass rather than shrinking: no MCP (held, with reasons), a standing approval pins the command and not its meaning, and Windows/Linux are unbuilt.

The same rot hit our own diagrams, and worse. We keep two: a system map of what the app is made of, and a test map of what protects it. Both were stale — and the test map's staleness was a bug in the tool that generates it. It cached a chain run with no way to tell the chain had changed, so every regeneration quietly re-described the previous suite. It claimed 73 steps and 34 clicks against a real 74 and 87.

That is L22 — a confident number over data that moved — living inside the tool that draws the picture of our tests. The cache is keyed to the test script now. And the system map's node count, which had been hand-typed and wrong for a week, is derived from the arrays.

Where this leaves the thesis

The first post argued that Cline's 8M developers came from trust ramps, context hygiene, and extensibility, and that Yee should learn from all three.

Building it, the trust ramps mattered most — and not because of the feature list. They mattered because every one of these lessons is the same lesson in a different costume: the system stated something that wasn't true. A modal that contradicted itself. A test that couldn't fail. A price that was invented. A trace that dropped its own history. A roadmap that listed shipped work as pending.

None of those are hard problems. They're all cheap to fix once seen. The expensive part is seeing them, and the only reliable way we found is to make the software prove its claims to something that doesn't take its word for it — a mutation, a click, a marker recording what got dropped.

That's what the remaining 30 rows are being judged against now. Not "does Cline have it."

Yee is a local-first AI code editor. The 81-row grid, the 74-step test chain, and both maps live in the repo — the test map is a standalone page you can open with no server: every step, what it protects in plain English, and the incident that caused it to exist.

Date: 2026-07-29 Source: cline.bot + docs.cline.bot + github.com/cline/cline (CHANGELOG through v4.0) + 2025-2026 reviews/HN — verified by a 10-agent orchestrated sweep (3 Cline researchers, 3 Yee code readers, 3 gap-analysis lenses, 1 completeness critic). All Yee claims below are grounded in code actually read this session (file:line cited). Status: ✅ Research complete — roadmap awaiting Ty's ranking pass Standing constraints honored: ⛔ no paid runs anywhere in this plan (every metric proposed comes from users' organic usage on their own keys) · no tab-completion · build ON existing modules, never rebuild · byte-stable system prompt (cache invariant) is inviolable · safety net sweeps everything, always.

TL;DR

Cline won 8M+ developers with three moves we should study: (1) trust ramps — Plan/Act hard-blocked modes, per-category approvals, checkpoints-in-the-chat — that let users dial autonomy up gradually; (2) context hygiene as first-class UX — Focus Chain, visible auto-compact, /newtask handoffs — their answer to long-run drift; (3) repo-resident extensibility — .clinerules, workflows, MCP — config that rides in git and spreads team-to-team. Their #1 documented churn driver is cost blowup despite transparency ($50–200/day team runs; the spend-limit UI came embarrassingly late, v3.78) — which is exactly the surface where Yee is structurally ahead (cache-aware pricing, 150k/300k budget gates, $0 local compaction).

Where Yee is ahead (double down, market it): the shadow-git DAG with real merge/rebase/safety-net (Cline's shadow git is branch-per-task restore; it's disabled entirely in multi-root workspaces), preview-anchored verification (error scrape → fix loop → Lighthouse → pre-push deploy fact-check), cost honesty with gates not just meters, and the farm→forge→split-test measured-tuning loop Cline has no equivalent of.

Where Cline is ahead (the gaps to close): recovery lives in a separate canvas instead of inline in the chat; our loop dead-ends silently at the iteration cap; compaction/degradation is invisible; easy mode has no read-only mode and silently overrides the user's auto-apply preference; approval grants are exact-string and session-only (and concurrent approvals race — a real bug); nothing we read rides in the user's repo (no rules files, no workflows); the model picker is a hardcoded 16-model list despite the OpenAI-compat plumbing existing; and there's no MCP story at all.

1. What Cline is (verified inventory, compressed)

Positioning (their words): "transparency over convenience, developer control over vendor lock-in, full model power over margin optimization, open source over proprietary black boxes." Apache 2.0, BYO-key, 40+ providers incl. local (Ollama/LM Studio) and OAuth piggybacking on ChatGPT/Claude Code subscriptions. $32M raised July 2025; enterprise tier Oct 2025; ClinePass ($9.99/mo open-weight models) June 2026.

Core loop features (all shipped, VS Code + CLI):

• Plan & Act modes — Plan is hard-blocked from edits/commands (structural, not behavioral); history carries across the flip; optional per-mode model pairing. The single most user-cited trust feature; Cursor and Copilot copied it.
• Checkpoints via shadow git — commit after every tool op; bookmark in the chat with Compare + Restore; three restore scopes (Restore Files / Restore Task Only / Restore Files & Task — i.e. they can rewind the conversation too, which we can't). Branch-per-task since v3.6. Disabled in multi-root workspaces; slows on huge repos.
• Auto-approve + YOLO — per-category toggles (reads/edits/safe-commands/all-commands/browser/MCP), model-assigned requires_approval per command, OS notifications when blocked ≥30s. YOLO = everything approved (v3.31, enterprise-disableable).
• Focus Chain (v3.25, default-on) — auto-maintained markdown checklist re-injected every 6 messages, n/m progress in the task header. Their fix for long-horizon drift.
• Auto Compact (v3.25) — visible summarization tool-call with its cost shown near the context ceiling; manual /smol; documented fallback to rule-based truncation. Their answer to "5M-token task in a 200k window."
• Slash commands — /newtask (package progress+plan+files into a fresh clean-context task), /deep-planning (silent explore → questions → implementation_plan.md → spawn fresh implementation task), /newrule, /smol, /reportbug.
• Diff editing — replace_in_file with order-invariant apply, continuously tuned against a public evals/diff-edits harness; they credit >10% gains to treating diffEditSuccess as the loop's core health metric. v4.0 moved toward apply_patch/editor tools.
• Loop detection + auto-retry with backoff (v3.76) and an experimental completion double-check pass (v3.58): every hard limit has a continuation ramp.

Extensibility (the ecosystem layer):

• .clinerules/ — repo-resident, version-controlled, team-shared rules; per-file toggle UI; conditional path-glob frontmatter; auto-detects .cursorrules / .windsurfrules / AGENTS.md (competitors' installed base = their onboarding funnel).
• Workflows — markdown files → /name slash commands, injected once as explicit_instructions (no persistent prompt tax); "generate a workflow from what we just did."
• Skills (v3.48) — progressive disclosure: ~100-token metadata always, <5k loaded on activation.
• Hooks (v3.36) — PreToolUse/PostToolUse scripts, fail_open/fail_closed. Notably: .clineignore is being deprecated because advisory filters got bypassed (mentions, shell) — replaced by hook-enforced runtime blocking. Enforce at the boundary or don't bother.
• MCP — STDIO servers, per-tool autoApprove arrays, in-product marketplace (Feb 2025), routed through the same approval system as built-ins.
• Subagents — read-only research fan-out (use_subagents) keeping the main context clean, per-subagent cost rolled into the task total.

Breadth we are explicitly NOT chasing (see §5): CLI/TUI, Kanban (parallel agents on worktrees), Agent Teams, Slack/Discord/Linear connectors, cron scheduling, enterprise SSO/governance, dictation, marketplace.

Their documented weaknesses (as valuable as the features):

1. Cost blowup — the persistent 2024-2026 complaint: $52–187/mo typical, "$50-200/day in team settings," 9× a Cursor seat; spend-limit UI only in v3.78 (Apr 2026). Transparency ≠ affordability. Gates matter.
2. Context regressions — the no-throttle philosophy produced token churn / context-loss regressions (issue #5616; users restarting every ~10 messages). Our cache-stability discipline is the countermeasure they lack.
3. No autocomplete + rougher polish — structural, acknowledged in most 2026 reviews. (Irrelevant to us: tab-completion is explicitly not our priority.)
4. Forkability — Roo Code forked them, iterated faster, siphoned power users through 2025 (archived May 2026). The durable moat was the eval-tuned harness + steadiness, not any copyable feature. Our equivalent moat is the measured substrate (farm→forge→split-test on the user's own keys) — any sharing feature must carry that discipline, never bypass it.

2. Lessons learned

Where Yee is AHEAD — double down, don't dilute

• L1. The checkpoint DAG is a marketable differentiator. Cline's whole autonomy thesis rests on "checkpoints drop the cost of a mistake to near zero" — and our engine is stronger where it counts: real merge/rebase, safety-net commits that sweep everything (incl. gitignored/stranger files), ignored-clobber guards, authored-only push staging ([lib/checkpoints.js](../building_block/editor/lib/checkpoints.js), [app/branchGraph.js](../building_block/editor/app/branchGraph.js), [app/pushSelect.js](../building_block/editor/app/pushSelect.js)). Cline's checkpoints are disabled in multi-root workspaces — an easy honest marketing line for us. One caveat where they lead: their three restore scopes include conversation rewind; ours is files-only (→ CL-01).
• L2. Cost honesty with GATES is the brand working. Cline proves per-loop cost display alone doesn't retain users — the churn came anyway, and the gate came late. Our cache-aware per-token-type pricing, ⚡cache-hit%, 150k soft / 300k hard per-turn pause with keep-going/stop ([app/synthwave.js](../building_block/editor/app/synthwave.js) BUDGET/budgetGate), and $0 local-mlx compaction are the preemptive version of their retrofit. Rule going forward: every new surface gets metered and gated on day one (MCP calls, compat endpoints, workflows) — and when pricing is unknown, show tokens + an explicit "$ unknown," never a fabricated figure.
• L3. Preview-anchored verification is our lane. Cline verifies via terminal watching + a heavyweight browser tool + an experimental double-check pass; our chain (noise-filtered preview error scrape → fix-in-chat through the same synthSend path → reverify → real Lighthouse pills → two-tier pre-push deploy fact-check that caught a real production failure) verifies at the surface the user ships. Nothing in their inventory has the pre-push deploy fact-check. But the critic's catch stands: this lane currently only covers web repos (→ CL-11).
• L4. Byte-stable-system-prompt discipline is a moat. Cline's context-regression record is what happens without it. Our invariant — EASY_QUALITY_PROMPT byte-identical, volatile context rides user turns, repo map pinned in the turn-1 pair surviving eviction, <30% cache-read tripwire — is the stricter, more honest version of their rules/skills tiering. Every feature in §4 injects via the turn-1 rider or follow-up prompts, never the system prompt. That's the design contract.
• L5. The measured substrate is the unforkable part. Promotion-requires-measured-score (on the user's own keys) is the discipline Cline's marketplace lacks and the reason a fork can't take the moat. Shared/imported anything arrives unmeasured and only earns default status through the user's own measured runs.

What Cline teaches us — the gaps

• L6. Never dead-end silently; every hard limit needs a visible exit ramp. Cline pairs every ceiling with a continuation (auto-compact, retry-with-backoff, loop detection, /newtask). Our runAgentLoop exhausts MAX_ITERATIONS=10 and just… returns — possibly with an unexecuted `tool block rendered as prose — even though we already built exactly the right UX for the spend limit. (→ CL-02)
• L7. Recovery must live where the panic happens — inline in the chat. Cline puts Compare/Restore bookmarks on every tool op in the conversation. Our stronger engine hides behind a separate canvas and a small 🛟 toggle — and our own 2026-07-18 "unrecoverable files" scare was explicitly a discoverability failure while the data sat safe in the shadow repo. An undo engine users can't find at panic-moment is functionally no undo engine. (→ CL-01)
• L8. Show the machinery: compaction, degradation, and dial-turns are trust surfaces. Their auto-compact is a visible tool call with a cost; our _capTurnsCompacted distills invisibly and — when local mlx is down — silently drops the conversation middle with no note. Also: easy mode force-calls setAutoApplyEdits(true) over the user's persisted preference with zero disclosure. The honest-instrument posture must cover internal state, not just dollars. (→ CL-06, CL-05)
• L9. A hard-blocked read-only mode is an onboarding ramp, not a power-user nicety. Plan/Act is Cline's most user-cited trust feature because the block is structural. We already have persisted edit/plan/auto behavior modes as "the canonical source of truth" ([app/agentLoop.js:1966](../building_block/editor/app/agentLoop.js)) — but they're main-editor only; the flagship easy-mode surface always runs full-act, force-auto-apply. First-run experience = maximum autonomy, no ramp. (→ CL-05)
• L10. Approval policy should key on what a command IS, not its exact bytes — and concurrent approvals must queue, never supersede. Their model-assigned requires_approval + persistent per-category toggles vs. our exact-string, session-only allowlist that re-prompts on one-char variation (training users to click through). Worse, requestCommandApproval denies a pending modal when a new one arrives — at PLAN_CONCURRENCY=3 an orchestrated step's command can be silently denied. That's a bug, not a preference. (→ CL-03, CL-04)
• L11. Long-horizon reliability = a persisted north-star artifact re-injected on a cadence. Focus Chain, mechanically: checklist, re-inject every 6 messages, n/m in the header. We have the artifact machinery (agentPlan checklists, steeringQueue's proven single-drain-point injection) and re-inject no task artifact (the repo map's MAP_REFRESH_EDITS drift refresh is the only rider refresh). (→ CL-08)
• L12. Context handoff is a first-class loop operation. /newtask forks the task state, not just the tree state. Our fork moves the workspace to a sha but the new chat inherits only the repo map — the accumulated understanding, which our local distiller could package for $0, is discarded exactly when it's most valuable. (→ CL-09)
• L13. Repo-resident config is the unit of ecosystem propagation. Rules/workflows that ride in git spread team-to-team; honoring competitors' formats (.cursorrules, AGENTS.md, CLAUDE.md) converts their installed base into your onboarding funnel free. Everything we read today lives in SQLite/localStorage — personal and machine-local; repos already carrying AGENTS.md are silently ignored. (→ CL-07, CL-13)
• L14. Extension tools must enter through the one choke point. Our safety-and-honesty spine (approvals, s.activity trace → PR bodies, farm capture, addTok metering) keys on the fixed tool vocabulary dispatched in executeTool. Any tool path that bypasses it forks the safety story and starves the meter/farm/PR surfaces. That's the MCP integration contract. (→ CL-16)
• L15. Enforce exclusion at the tool-call boundary, fail closed. Cline is deprecating .clineignore because advisory filters were bypassed. We currently have no exclusion mechanism at all — every repo file is readable, mappable, and farmable into the SFT warehouse, .env included. We get to skip their advisory mistake entirely. Scoping rule: ignore governs model access + training capture, never the safety net's recovery sweep. (→ CL-17)
• L16. Edit-apply success is the loop's core health metric — and its failures are free training data. They publish an evals/diff-edits harness and credit >10% gains to it. We already persist the per-tool-call ok bit in s.activity and farm only successful edits — the failure half of exactly this signal is discarded today, and it costs zero paid runs to keep (it's organic usage). (→ CL-12)
• L17. Provider breadth is the no-lock-in adoption wedge, and most of it is one generic OpenAI-compatible endpoint. (Not all — their subscription-OAuth piggybacking is a separate mechanism a compat endpoint can't reach; that stays a watch item.) Our mlx: path already speaks OpenAI-compat HTTP to a configurable URL; the lockout is just the hardcoded 16-model picker and pricing table. (→ CL-14)
• L18. Community rituals become product. Memory Bank is a docs-canonized pattern where the frontier model re-reads six markdown files every task — a recurring token tax. We can ship the same continuity as a near-zero-cost built-in because we already persist the raw material (sessions, s.activity, s.authored, tok, DAG) and own a local distiller — PR-body synthesis already proved zero-token composition works. (→ CL-22)

3. The backlog, ranked by need

Legend: P0 = trust/correctness gap users hit now · P1 = high leverage next · P2 = valuable, after P1 · P3 = someday/watchlist. Effort S/M/L/XL. Every item builds on named existing modules; none needs a paid run; all injection is cache-safe (rider/user-prompt only).

P0 — close the trust gaps now

CL-01 · Inline turn-restore chips in the chat stream — Every assistant message whose turn produced edits gets a hover affordance: "⎌ restore to before this turn" + "view changes." Restore calls the existing restoreTo (already safety-commits live state, re-places strangers, guards ignored clobbers) against the turn's baseline sha the DAG already stores; confirmation is the existing swap system-note with one-click undo; "view changes" reuses the branch-graph node's lazy diff card. The canvas stays the power surface; this puts the panic path where the panic happens (L7 — and the 2026-07-18 scare is the receipt). Restore scopes (critic catch, and the one axis Cline leads on): after a files-only restore the conversation still "remembers" edits that no longer exist — v1 must at minimum stamp an honest system note into the transcript ("workspace restored to before turn N; edits after that point are no longer on disk") that also rides the model context; a full Cline-style "rewind conversation too" is a fast-follow once the note pattern proves out. Acceptance: click restore on a mid-chat turn → files revert, safety node appears in the canvas, note visible to user AND model, undo works — verified in the live app, not just tests.

CL-02 · Iteration-cap continue gate + loop detector — When MAX_ITERATIONS exhausts with apparent unfinished work (trailing unexecuted `tool block, or edit-intent detected but unapplied), pause with the exact budgetGate UX: "hit the 10-round safety cap" + keep-going (buys another leg) / stop. Render any unexecuted trailing tool block as an honest "planned but not executed" block, never prose. Companion (critic catch): a repeated-tool-call loop detector (same tool + args N times, or no-progress heuristic) — without it, "keep going" can repeatedly fund an infinite loop; with it, the gate message says "the agent appears to be repeating itself" so the user's choice is informed. (L6) Acceptance: a deliberately over-long task pauses with the gate instead of ending silently; a synthetic repeat-loop trips the detector message.

CL-03 · Approval FIFO queue (straight bug fix) — requestCommandApproval currently denies a pending modal when a new request arrives; orchestrated plan steps at PLAN_CONCURRENCY=3 can race and silently lose commands. Queue FIFO. Ship independently of CL-04 — it's a correctness fix, not a feature. (L10)

CL-07 · Repo rules rider — At chat creation, discover rules at repo root in priority order .yee/rules.md → AGENTS.md → CLAUDE.md → .cursorrules → .clinerules/.md, concatenate with per-file caps, inject as a labeled section of the turn-1 user prompt beside the repo map (pinned pair survives eviction; refresh on the existing MAP_REFRESH_EDITS drift path when the file hash changes). Header pill names which file loaded, with a toggle-off — the user sees exactly what rides the prompt. Never touches the system prompt (L4). Honoring the competitor formats converts Cursor/Claude-Code-convention repos into instant-onboarding repos (L13). Verified: zero rules-file support exists today. Also builds the shared repo-config loader that CL-13/16/17/22 reuse. Acceptance:* drop an AGENTS.md into a repo → pill appears, instruction observably honored in output, cache-hit% unchanged across turns (the tripwire is the regression test).

P1 — control, continuity, and the first open doors

CL-04 · Persistent classified command allowlist — Third approval tier: "always allow <class> in this repo," keyed on the existing classifyCommandPurpose class + normalized command family, persisted per-repo in chatStore. Canonical constraints (the critic forced this to be explicit): the gateway evaluateTerminalSecurity blocklist always wins; destructive classes (delete, install, git-mutating) are never offered for standing approval; the tool block shows which tier auto-approved ("auto-approved: run-tests allowlist") so grants stay auditable; a header chip lists/revokes grants. Exact-string session tier stays. (L10)

CL-05 · Plan/Act toggle in easy mode + honest Act disclosure — Surface the already-persisted behavior modes as a two-state header switch. Plan: executeTool hard-blocks EDIT_TOOLS + run_terminal with a structured "planning mode — read and propose" tool result (the existing rejection-as-tool-result pattern, so the model adapts instead of erroring). Act: current behavior, but with the missing disclosure line: "Act auto-applies edits — every turn is checkpointed and restorable" — fixing the silent setAutoApplyEdits(true) override (L8, L9). Optional fast-follow: per-mode model memory via the existing per-conversation picker (cheap model for planning, user's choice, user's key). Acceptance: in Plan, zero file mutations across a full multi-iteration turn (checkpoint diff is empty); flip to Act mid-conversation carries history; disclosure visible on first Act turn.

CL-06 · Visible compaction + manual compact + context meter — (a) _capTurnsCompacted emits over the existing usage-normalization channel; renders a one-line system note: "condensed 12 older turns into a summary (local model · $0)." (b) When mlx is down and plain eviction drops the middle: say so — "older turns dropped without summary — local compactor offline." (c) "Compact now" affordance. (d) Small context-fill pill beside the token meter (last input tokens vs model window, from a per-model window table next to MODEL_PRICING — same honest-rough posture, and note the shared maintenance debt: neither table has a reconciliation mechanism; keep them adjacent so one update pass covers both). Extends honest-instrument from dollars to context state (L8). Notes ride the chat surface, never the prompt.

CL-08 · Task ledger (focus-chain-lite) — Turn-1 drafts a ≤10-item checklist for multi-step asks; the loop appends it compactly to follow-up prompts every Nth iteration via the steeringQueue's proven single-drain-point; model checks items off; header shows n/m. Persisted with the session so it survives eviction and restarts; rides follow-up prompts only (L4, L11). Reuses the agentPlan checklist schema — this is prompt mechanics, not new machinery.

CL-09 · Fork-with-context handoff — "Hand off to fresh chat" node action on the branch graph: distill the conversation via the local compactor ($0) into task-state bullets, carry s.authored + repo map + ledger, run the existing fork flow with the new session pre-seeded. The handoff is itself a DAG node, so lineage stays visible. mlx-down degrades to extractive bullets with an honest note. (L12)

CL-10 · OS notifications — Electron Notification (window-unfocused only): approval modal opened (loop is blocked on you), budget/iteration gate paused (spend frozen), long turn finished, Lighthouse queue done/stalled. Click focuses the window; one quiet-mode toggle. Today a user who tabs away leaves the agent frozen at a modal. (Cline fires these at approval-needed and 30s-running marks.) The fix-queue trigger ships only after the queue's own end-to-end click-through — it's marked untested on the system map; don't build notification UX on an unverified sequencer.

CL-11 · Easy-mode verification for non-web repos — The critic's best catch: our verification moat (L3) covers only running web apps; for CLIs/libraries/Python repos, easy mode verifies nothing after an edit. Bring projectVerify/lintCheck into the easy-mode turn lifecycle: after the turn's edits, run the repo's detected check (typecheck/test/lint) and surface pass/fail as a chip; failures offer the same fix-in-chat path as preview errors. Do the debt fixes on the way in — they directly cause wasted paid fix-turns: package-manager detection (npm hardcoded breaks pnpm/yarn), and lintCheck's new Function() false syntax errors on any import/export (most modern source). Scope v1 to JS/TS + Python detection; cargo/gradle later. Acceptance: in a pnpm TS repo with no dev server, an easy-mode edit turn ends with a real typecheck verdict chip; no false syntax errors on ESM files.

CL-12 · Edit-apply success meter + farm negatives — Aggregate persisted s.activity {t,p,ok} into per-model, per-edit-tool apply-success rates with always-visible n counts in the under-the-hood pipeline popup (upgrading the n=2-12 MODEL_BEST strings toward defensible numbers as organic n grows). Capture FAILED search_replace/multi_edit attempts as negative rows for the forge's negative-weighting pipeline — the failure half of the signal is discarded today (L16). Zero paid runs by construction. Prerequisites (critic): fix s.activity honesty first — the 400-entry cap and 80-char path truncation currently lose early activity with no "and N more" marker, which would silently understate this meter, the receipt line, and PR bodies. Also label truncation-induced mismatches (model edited text it never fully read due to 8000-char read caps) so tool-caused failures don't pollute model-attributed negatives.

CL-13 · /workflows — Markdown files in .yee/workflows/ (repo) + a userData global folder become /name commands with autocomplete in the prompt box; on invoke, wrapped in a short explicit-instructions preamble and dispatched through synthSend's existing forced-prompt parameter as one message. The dispatch engine already exists — fixPreviewErrors, fixPythonErrors, deploy fix-in-chat, and the Lighthouse queue are four shipped internal workflows with no user authoring surface. Plus zero-token "save as workflow" from the current turn's prompt + s.activity tally (authoring from data in hand, no model call). (L13)

CL-14 · Generic OpenAI-compat provider — compat: model prefix resolved to a user-configured base URL + optional safeStorage key, reusing the mlx path's OpenAI-compat client. Settings CRUD: name, URL, optional user-entered $/Mtok. The meter always shows tokens; $ only when the user priced it — explicit "$ unknown — set a price" otherwise, never fabricated (L2, L17). Opens OpenRouter/Groq/DeepSeek/LM-Studio/self-hosted without shipping 40 integrations. (Subscription-OAuth piggybacking — Cline's other adoption lever — is not reachable this way; it stays on the watchlist, see §5.)

CL-15 · Renderer↔main version handshake — The critic generalized our "pervasive" restart-required gotcha (push, staging, checkpoints, chatStore columns — and CL-06/16/18/21 each add new IPC): a build stamp exchanged at boot; mismatch shows one "Yee needs a full restart" banner everywhere, instead of per-feature tripwires and mystery staleness. Small, and it should land before the IPC-adding features.

P2 — open doors, wider capture

CL-16 · MCP client v1 — STDIO-only, local servers from .yee/mcp.json + global config; main-process lifecycle patterned on devServer.js; tools mirrored into the text-protocol vocabulary, frozen at chat creation (same fixed-at-creation cache invariant as language groups). Every call dispatches through executeTool behind a new gate class: default always-ask modal labeled server:tool, per-tool session allow, results through compactResult, every call in s.activity (PR bodies honestly show third-party tool use) and metered (L14, L2). Manual JSON config + 1-2 documented example servers; explicitly no marketplace. Default-off until the approval UX is click-through verified. Sequencing (critic): the nativeTools.js mirror path is marked untested on the system map — v1 rides the text-vocabulary path only; native mirroring waits for native mode's own verification. Retires the README's honest "no plugin system" gap.

CL-17 · .yee/ignore enforced at the boundary — Gitignore-syntax exclusion (secret-pattern defaults on) enforced inside executeToolDirect for read/grep/list/write, in repoMapFor, and in captureEdit so excluded paths never enter the SFT warehouse. Blocked access returns a structured "path excluded" tool result so the model reroutes. The safety net still sweeps everything — ignore governs model access and training capture, never recovery (guard-test this; the 2026-07-18 scare makes sweep completeness sacred). Skips Cline's advisory-phase mistake entirely (L15). Header disclosure when active.

CL-18 · Resume interrupted turn — Tiny in-flight marker per session; on load after a mid-turn death, an honest note + one-click resume building a continuation prompt from the s.activity tail + dirty-diff against the turn's baseline checkpoint, through the normal synthSend path.

CL-19 · Per-turn receipt line — Collapsed one-liner in the message footer: tokens in/out, cache-hit%, ~$, N tool calls (M edits), files touched — all from data the turn already counts, formatted to match the PR body's stat line so the live chat and the PR reviewer read the same numbers (the instrument agreeing with itself across surfaces).

CL-20 · fetch_web tool — Text-based HTML→markdown fetch through the executeTool choke point: gated (approval class "network read"), metered, compacted. The light alternative to a browser tool; Cline ships this (v3.48) and we have no web retrieval at all. No search API dependency in v1 — URL fetch only (search needs a provider and is a separate decision).

CL-21 · Multimodal attach — Drag-drop/paste an image into the chat (mockups, bug screenshots — a core easy-mode use case the critic flagged). Rides the existing screenshot→chat vision path incl. the honest token-estimate chip (⌈w/28⌉×⌈h/28⌉); non-image files deferred.

CL-22 · Portable .yeepack bundles — Export a stream/pack (wd/rm/pa + params + lineage) as versioned JSON, committable to .yee/packs/; import shows as a fork labeled "imported — unmeasured", excluded from default until the user's own measured promotion clears it (L5). Author scores display as advisory only; local promotion always re-measures. Marketplace-lite with the discipline Cline's marketplace lacks; pairs with the data library's export-bundle direction.

P3 — someday / watchlist

• CL-23 · Zero-cost repo memory brief — "Previously in this repo" composed from persisted sessions/activity/authored/DAG, optionally distilled by local mlx; rides the turn-1 rider; user-visible + toggleable. Memory Bank's value without its per-task frontier tax (L18 — and per the critic: the honest claim is "zero-config built-in over data we already persist," not "structurally impossible for Cline").
• CL-24 · Read-only research subagents — fan out exploration to read-only agents with own budgets, results summarized into the main context. Real value for repo exploration burning the 24-turn window, but heavy; after ledger + handoff prove out.
• CL-25 · Skills-style progressive disclosure — tier-3 instruction packages loading on demand via the user prompt (cache-safe). After CL-07/13 establish the config-loader + authoring habits.
• CL-26 · General pre/post-tool hooks — the single choke point makes this cheap someday; CL-17 is the concrete instance first. fail_closed semantics from day one.
• Watchlist, no commitment: subscription-OAuth piggybacking (adoption lever, but ToS/complexity-fraught; revisit if BYO-key friction shows up in real usage) · Python Space/Jupyter completion (Cline ships cell-granularity notebooks; ours is PARTIAL — explicitly deferred, not forgotten, pending Ty's call) · local-whisper dictation · edit-and-regenerate of past messages.

Explicitly NOT pursuing (decided, with reasons)

• Tab completion — Ty's standing call; Cline's reviews confirm agent-does-the-work can win without it.
• CLI/TUI, Kanban, Agent Teams, connectors (Slack/Discord/Linear), cron, enterprise SSO — Cline's breadth wave serves their platform play; our thesis is depth on the local-first editor + Forge ecosystem.
• YOLO mode as a labeled feature — easy mode already auto-applies within a checkpointed sandbox; what we need is the disclosure (CL-05), not more throttle.
• MCP marketplace — manual config first (Cline's own maturity order); a marketplace without the measured-substrate discipline would dilute L5.
• Anything requiring a paid eval run to build or justify — standing rule; every metric above is organic-usage-derived.

4. Suggested milestones (sequencing, not new scope)

• M1 — Trust the loop (all S/M, ~quick): CL-03 (bug), CL-02, CL-01, CL-07, CL-10, CL-15. Outcome: no silent dead-ends, recovery inline, rules honored, walk-away-able runs.
• M2 — Long-run control: CL-05, CL-04, CL-06, CL-08, CL-09, CL-19. Outcome: onboarding ramp + honest machinery + context continuity.
• M3 — Verify everywhere, measure honestly: CL-11, CL-12 (+ s.activity honesty fix), deploy-check manifest reuse (Cargo/poetry via pushSelect's LOCKFILE_PAIRS — cheap, critic-flagged), Lighthouse route context. Outcome: the verification moat covers non-web repos; apply-success becomes our public-grade health metric.
• M4 — Open doors: CL-13, CL-14, CL-21, CL-20, CL-17. Outcome: repo-resident ecosystem + any-endpoint BYOK, exclusion enforced before capture widens.
• M5 — Ecosystem bets: CL-16, CL-22, CL-23, then P3 re-rank. Outcome: MCP behind the spine; sharing with the measured-substrate discipline.

Candidates for the 🗺 PRIORITY TRACKER (Ty's call): CL-01, CL-02+03, CL-07 as the P0 block; CL-11 as the thesis-completing P1.

5. Standing engineering contracts every item above obeys

1. Cache invariant: nothing ever mutates the byte-stable system prompt; injection = turn-1 pinned rider or follow-up prompts; per-chat vocabulary freezes at creation. The <30% cache-read tripwire is the regression alarm.
2. Choke-point contract: every tool — built-in or MCP — dispatches through executeTool, so gates, s.activity, farm, and the meter see everything.
3. Safety-net completeness: the shadow-git sweep captures everything, always; no ignore/exclusion feature may narrow recovery. Guard-tested.
4. Honest degradation: when a subsystem is down or a limit is hit, say so in the chat surface; never silently degrade (compaction, iteration caps, unknown pricing, truncated activity).
5. Verify at the product surface: each shipped item's acceptance criterion is a live-app click-through, not a green unit test (SHIPPED_PACKS rule). Items building on system-map untested nodes (native tools, Lighthouse fix queue) sequence that verification first.
6. No paid runs: all metrics are organic-usage-derived on users' own keys; imported/shared config is unmeasured until the user's own promotion clears it.

Appendix: primary sources

• https://docs.cline.bot — core-workflows (plan-and-act, checkpoints, using-commands, working-with-files), features (auto-approve, auto-compact, focus chain, multiroot), enterprise docs
• https://github.com/cline/cline — README + CHANGELOG (v3.1 → v4.0, Jan 2025 → Jun 2026)
• https://cline.bot/blog — v3.25 (Focus Chain, Auto Compact, /deep-planning), v3.31 (YOLO, task header), v3.48 (Skills, websearch), v4.0 (SDK re-platform)
• Reviews/community 2025-2026: cost-blowup complaint record, context-regression issue #5616, Roo Code fork arc (archived May 2026), Plan/Act reception threads

A changelog for everyone following along. Last time, the whole project arc told the full story of YeeYee — a local-first, offline-capable AI code editor whose thesis is the agent does the work — and ended at roughly "feature-complete, validation remaining." This is what happened in the seventeen days after that post. Short version: feature-complete became shippable, and dogfooding taught us more than any benchmark did.

TL;DR

• YeeYee is now a real Mac app. Signed, notarized by Apple, Gatekeeper-clean DMG. Double-click,

drag to Applications, done — no right-click-to-open ritual.

• The token meter now tells the truth. Cost estimates used to run ~7× high; they now match the

provider's own console to within pennies. A prompt-cache bug that was silently re-writing context on every call got found, proven by replay, and killed — one measured session dropped from $0.60 to about $0.10.

• The agent learned git manners. It commits only what it actually touched, fact-checks "will this

deploy?" before every push, does real rebases with a recoverable safety net, and writes pull-request descriptions that tell the story of the conversation that produced them.

• The live preview got street smarts. Multi-app repos get a launcher, backend services are treated

as services (not broken web pages), port conflicts get diagnosed instead of "fixed," and Lighthouse scores sit in the chat header with a one-click path to fixing them.

• We stopped running paid internal evals — and shipped the measurement tooling to users instead.

More on why below; it's the most honest decision in this post.

1. Shipping a real Mac app (July 12 → 25)

The arc post ended with an editor that worked great — on the machine it was built on. Getting from there to "a stranger double-clicks a DMG and it just works" turned out to be its own project.

• July 12 — a one-command installer build, a launch-readiness audit that surfaced four

ship-blockers (all fixed), and a safety-hardening pass over the surfaces that could actually hurt a user: file edits, secrets handling, the terminal, and the embedded webview.

• July 21 — a turnkey code-signing and notarization pipeline: one command preflights the

credentials, signs with hardened runtime, notarizes, staples, and verifies the result.

• July 25 — the first fully notarized DMG, verified accepted — Notarized Developer ID at both

layers. Plus the cosmetics a real app deserves: a proper app icon and a styled installer window instead of a bare dark rectangle.

Two hard-won lessons for anyone shipping an Electron app on macOS, because these cost us real days and the internet's answers were thin:

1. Apple's G2 intermediate certificate must live in a persistent keychain. Modern Developer ID

certs chain through an intermediate that macOS doesn't ship. With it missing, codesign fails intermittently — roughly four runs in five — which means one lucky green build proves nothing. One certificate import and the pipeline went thirteen-for-thirteen.

1. Notarizing the app is not notarizing the DMG. The standard tooling happily notarizes the .app

inside while leaving the DMG container itself unsigned — and Gatekeeper judges the thing the user actually downloads. Sign, notarize, and staple both layers, and verify both.

And then dogfooding the installed app immediately paid for itself: launched from the Dock, the live preview died with spawn npm ENOENT. Apps launched from Finder get the OS's bare PATH — not your shell's — so every developer tool the app shells out to is invisible. The fix (capturing the user's real login-shell environment at boot, the way the big editors do) closed the same hole in three other features that would have quietly broken for every beta tester.

2. The token meter now tells the truth (July 16 → 18)

If you've read the earlier posts, you know the standing rule here: no headline without a measurement. This month the measuring instruments themselves got audited, and the findings were humbling in both directions.

• Cost estimates ran ~7× high. The meter priced every token at one blended rate; real pricing

differs per model, per direction, and per cache status. Rebuilt with real per-model, per-token-type pricing and verified against an actual provider bill: a session the old meter called ~$25 prices to ~$3.50 — matching the console. Every dollar figure in the app (meter pill, per-conversation spend, usage history) corrected at once.

• The prompt cache was being wasted. Console data from a real session showed only 4% of input

tokens read from cache — nearly the whole bill was cache writes, paying the premium and never collecting the discount. A replay investigation proved the mechanism (a history-trimming step was shifting the message prefix on every call, invalidating the cache each time), and the fix took that session's measured cache reads to the ~85–95% range. Honest saving on that one receipt: about 6× — $0.60 down to roughly $0.10. That's a receipt, not a benchmark; your sessions will vary.

• Some telemetry was dead and nobody knew. Months of usage, zero rows of token history on disk —

the plumbing between the model provider and the storage layer was silently dropping usage data for an entire routing path, including local models. Now fixed, normalized at one boundary, and locked with tests so it can't silently regress. Lesson worth stealing: when a cost surface reads empty, suspect the plumbing before the storage.

Around the meter, the spend-control story rounded out: every conversation permanently carries its own token and dollar totals, the start screen shows a year-view usage tree, a spend circuit-breaker pauses runaway loops — and you can always stop. The send button becomes a stop button mid-run, and the budget pause now has a real "stop here" that keeps the work already done.

The chat header pill now reads tokens · cache-hit % · estimated $ at a glance, with a tripwire that warns if cache churn ever creeps back.

3. Git manners: the agent stops treating your repo as its property (July 21 → 25)

This is the section I'd most want other AI-editor builders to read. The blunt product feedback that kicked it off: the agent treated the entire working tree as its own. Every AI coding tool we've tried does some version of this. YeeYee now doesn't:

• It publishes only what it authored. The agent tracks which files it actually edited, and a push

stages exactly the selection you approve. Files changed outside the chat — by you, by another tool — survive branch swaps intact and appear at push time clearly labeled "changed outside this chat," unchecked by default. The safety-net checkpoints still capture everything (recovery keeps working); publication is deliberate.

• "Will this deploy?" is checked before you push. Born from a real scar: a mechanically-successful

push once shipped a dependency-manifest change without its lockfile, and a production site of ours served a stale build for two days while every signal looked green. Now an always-on, sub-second deployability check runs after every edit, a full local deploy build is one click away, and the push dialog blocks a known-broken push — with a mechanical, zero-AI-tokens one-click lockfile fix, because a deterministic problem should never cost you an agent turn.

• Real rebase, with a safety net. The visual branch canvas already had fork, swap, and merge;

now it has rebase — powered by git's actual rebase machinery underneath, not a hand-rolled replay. The originals stay pinned and recoverable, conflicts reuse the same three-way resolution UI as merge, and a half-finished rebase survives an app restart.

• A real-git health banner. People bounce between editors and terminals all day; none of them talk

to each other. YeeYee now warns — visibly, above the chat — when you're behind your branch's upstream, on a detached HEAD, or sitting on a rebase another tool left mid-flight. It never blocks and never auto-runs anything: the suggested fix is typed into the terminal for you to review and run.

• Pull requests that tell the story. A YeeYee-opened PR now carries what was asked, what the agent

did (a tool-by-tool tally), which files changed, and the full transcript collapsed at the bottom — synthesized from data already in hand, zero extra tokens.

The receipts, because process is the product here: this batch went through adversarial multi-agent review before ship, and those reviews caught 48 real bugs — including a rebase edge case that could have clobbered gitignored files and a stash sequence that could have stranded uncommitted work. Every one was fixed and got a regression test before a user ever touched the feature. Agent-found bugs are cheaper than user-found bugs; that's been true every single week of this project.

4. The preview grows up (July 15 → 19)

The live preview is the heart of the "agent does the work" loop, and it collected a month of street smarts:

• Multi-app repos get a launcher. A repo with several runnable apps now shows you the choices;

swapping the preview target is one click.

• A service is not a broken web page. Previewing a backend service or crawler as if it were a

website once produced a million-token agent no-op — the agent dutifully trying to "fix" a page that was never meant to render. Services are now recognized and presented as services. That single distinction deletes an entire class of wasted spend.

• Port conflicts are diagnosed, not "fixed." Dev servers that auto-hop to a free port are allowed

to; a genuinely stuck port is reported as an environment problem — naming which of your own running apps is holding it — and is never fed to the AI as a code bug to fix. (Previous behavior, confession: it once killed a healthy server one log-line before that server printed its URL.)

• Lighthouse in the chat header. Four real scores — performance, accessibility, best practices,

SEO — live next to your conversation, and any score below 100 is one click from a chat turn that goes and fixes it.

• Smaller but daily-life: the Python notebook's run button is now boringly reliable (plus an explicit

"clear output"), the element picker supports multi-select with color-paired references you can discuss in chat, and the model dropdown grew provider groups, sensible recommendations, and recents — including same-week support for the newest frontier models.

5. The honest-instrument decision: we stopped paying for our own evals (July 16)

The arc post ended with two features "code-complete, awaiting paid evaluation runs." Here's the update, and it's not the one we expected to write: those runs are on hold indefinitely — and that's the honest call.

The measured record earned it. Our per-language tuning lever demonstrably rescues weaker models — that result was real and statistically significant — but on frontier models the same lever was either noise or robustly cost more. More eval cells weren't going to change what we ship; they were going to burn money confirming a curve we could already see.

So the instrument shipped to the people it actually serves: split-testing is now the users' tuning surface. You run splits on your own keys against your own repo, and the same honest statistical gates we built for ourselves — minimum sample floors, bootstrap confidence intervals, replay guards — protect your measurements. A winning configuration gets promoted into your model dropdown for your repo's language group. Firsthand use of a launched product teaches more than another synthetic benchmark cell. The standing rule is unchanged: no headline without a measurement.

Two transparency features shipped in the same spirit:

• The system map — an interactive network diagram of the entire application (38 nodes, 56

connections), now embedded in the app itself, where every node honestly shows its status: shipped, partial, or untested. Most software hides that; we think showing it is the product.

• The Forge door — the live model-training dashboard (the mine → train → evaluate loop from the

arc post) is now a door on the start screen, not a separate tool you have to know about.

What's next

Beta DMGs are going into testers' hands. The newest git surfaces are bench-proven and headed for live click-through. Hard mode is visible on the launch screen behind a "coming soon" gate. And we'll keep publishing the methodology — the metering math, the statistics, the process failures — because the receipts are the point.

Previously: the whole project arc. YeeYee is software developed by dev3lop.com.

This one is for the re-run @yohjisakamoto described: raw provider usage objects, normalized fresh/cache-read/cache-write splits, measured vs estimated buckets, n≥5 with bootstrap CIs and tier stratification, wiring and meter checks as preconditions. That list is right, and it's more rigor than any public comparison of these tools has had. But every gate in it validates a single run's record. Nothing in it validates the sample — and the statistics you're about to compute assume something about the sample that agentic benchmark runs quietly violate.

A bootstrap CI treats your runs as independent draws from one distribution. In this domain they aren't, in at least five specific ways. Each of these cost us a wrong conclusion (or nearly did) before we found it, and none of them appear in any methodology writeup we've seen. Consider this the field guide we wish someone had handed us.

1. The cache couples consecutive runs

Provider prompt caches persist for minutes to an hour after a request. Run configuration A, then configuration B on the same task, and B inherits whatever prefix cache A just paid to write. Whichever arm runs second looks cheaper — regardless of the tool.

This is not a small term. In our own editor, cache state alone moved a session's bill ~6× with zero change to model, tool, or prompt (the cache-churn incident from the earlier post: 4% read rate where ~95% was expected, $0.59 of a $0.60 session in cache writes). When 96% of tokens are cached input, cache state is a bigger term than most of the tool effects anyone is hunting. A comparison that doesn't control it is measuring cache weather.

What we do about it:

• Interleave arms, never run them in blocks. A-A-A-A-B-B-B-B maximally confounds arm with cache state (and with time — see §2). Alternate or randomize.
• Decide a cold-start policy and record it. Either force cold (gaps longer than the cache TTL, or a distinct prefix per run) or give both arms the same warmup and drop run #1 from each.
• Keep fresh-input as its own column. With the read/write/fresh split recorded per run, you can compute the comparison both on total billed tokens and on fresh input alone. When those two disagree, cache coupling is the reason — and that disagreement is itself a finding.

2. A sweep is a time series, and the system drifts under it

An n≥5-per-cell, tier-stratified sweep takes days. Providers ship silent model updates on that timescale. Cursor ships harness updates on that timescale. Pricing tables drift on that timescale — our own cost readout was confidently 7× wrong for a while, and that was client-side drift; the provider-side kind you can't even read in a diff.

If arm A ran Monday and arm B ran Thursday, you compared two different worlds and labeled the difference "the tool."

What we do about it:

• Our measurement harness interleaves per step — each step is a complete model×variant comparison, so drift lands on both arms equally instead of on one.
• Record the model ID from the response, not the request. The request says what you asked for; the response says what actually served the run.
• Record harness and plugin versions in the run row. If either changes mid-sweep, the sweep has a seam — analyze the halves separately or start over.

3. Your sample is a mixture, and the mean of a mixture lies

Our noisiest model refused to produce a stable token number: a 6.4× range (4,091 to 26,358 tokens) on the same task, CV ~50%. The temptation is to call that randomness and average harder. It wasn't randomness. Reading the logged commands showed a distinct failure mode — the model used the terminal as its primary interface, spelunking the shell (cat, sed, find, grep, re-running the same build) instead of using its tools, and how deep it dug was the variance.

That means the sample wasn't one distribution. It was a mixture: clean runs and spelunking runs. And interventions act on the mixture, not the mean — the directive we shipped for this cut the mean −38% (n=12), but its real effect was killing the catastrophic tail: worst case dropped 47%, from 34,154 to 18,159 tokens.

What we do about it:

• Classify every run before averaging: completed clean / wandered / retried / failed. The classification comes from the logs, and it's cheap.
• Report the mixture shift next to the mean shift. A tool that converts wanderers into clean runs can leave the median untouched and still be the most valuable thing you tested. Averaged blindly, it looks like nothing.
• Work in log space (multiplicative costs, heavy tails) — the log-ratio bootstrap from earlier in this thread exists because one 26k-token run should not own your mean.

4. Variance is a result, not an error bar

The most robust finding in our whole efficiency investigation wasn't a mean. On one reasoning model, our intervention read −32% mean tokens — but the number that replicated cleanly was the variance collapse: CV 75% → 22%. Runs went from swinging 6× to landing in a tight band. (An earlier n=2 probe of the same effect read −63%; the direction was robust, the magnitude was n-dependent. Treat your own early magnitudes the same way.)

Two tools with identical means and 3× different spread are not equivalent products. The tight one you can budget; the wide one hands you a surprise bill some fraction of the time. If the re-run reports only means and CIs, this dimension is invisible.

Report per-arm CV (or IQR) as a first-class result. A variance change with a flat mean is worth publishing on its own.

5. The verifier is part of the instrument

Billed-usage-per-verified-success puts a verifier inside the measurement loop, and the verifier has its own failure modes. Two of ours:

• Metric mixing. Our headline eval score climbed impressively — and decomposition showed it climbed because apply-rate climbed (did the edit apply at all), not correctness. One score was silently blending "produced output in the right shape" with "produced the right output." De-mix before you trust a denominator: applied? passed real tests? honest about what it did?
• The too-good score. A 100% result turned out to be train/eval leakage — the system had effectively seen the answers. Your version of this is subtler: repeated runs of the same task pool, with caching and retrieval in the loop, can "learn" the benchmark across repetitions. A score that improves across repeats of an unchanged configuration is a leak detector firing.

And if verification is an LLM judge: pin its version, record it per run, and accept that its variance is now inside your CI. Given the coupling point from earlier in this thread (compression moves usage and pass rate), a drifting verifier makes any usage delta unfalsifiable — you can no longer say whether the bill moved or the bar did.

What this adds to the run row

The record schema this thread converged on, plus the sample-level columns:

• interleave position / arm order index, and timestamps
• cold/warm cache flag (and the fresh-vs-total comparison both ways)
• model ID as reported by the response, harness + plugin versions
• run classification (clean / wandered / retried / failed), from logs
• verifier ID + version, with de-mixed outcome fields

The gates already agreed in this thread validate each row. These columns validate the set of rows — which is what the bootstrap actually resamples.

None of this is theoretical caution. Every section above is a conclusion we nearly shipped wrong: the warm-arm comparison, the drifted sweep, the averaged mixture, the buried variance win, the mixed metric. The gates caught none of them, because the gates audit records and these live between records. When your numbers land, this is the layer I'd most like to compare notes on.

A quick note, whenever you use any contact page, they might be able to track you. Today, we are tracking the IP address of any spammy signals, and then we begin communicating with that hosting company to understand more. We do this because it's exciting and fun. It's the game we must play online, and we accept your request to play. See you on the digital battle fields.

The reason j7 was created to be a honey pot for attention, and get the attention off of the priority solutions. This is saving a ton of time/money having to support individual websites under attack. Welcome to 2026. Hackers have ai, and so do we.