What building the (code editor) roadmap taught us that writing it couldn't

A follow-up to Cline lessons learned + feature roadmap for Yee easy mode.

Last time I published a competitive study of Cline, eighteen lessons, and a ranked backlog: 26 features, broken into 81 steps, each scored for complexity, sequenced across five milestones. It ended with "research complete — roadmap awaiting a ranking pass."

We did the ranking pass. Then we built it.

Where it stands: 51 of 81 rows shipped, 111 of 188 complexity points. Every row through the same gate — a hermetic test guard, the full chain green, an adversarial review, and a commit that says what it did. The test suite is now 74 steps and 2,510 assertions and costs nothing to run: no model call, no network, so it runs on every change.

That's the boring part. The interesting part is that executing a roadmap teaches you things writing one cannot, and three of those things were uncomfortable enough to be worth a post.

L19 — The roadmap was wrong in one specific, dangerous place

Row CL-04 was "classified command allowlist." The idea: instead of approving npm test over and over, approve the class of command once and stop being asked. Cline does something like this. It's an obvious quality-of-life win. I ranked it P1 and specced it in four steps.

Before writing any code, we ran three independent threat models against the design and had them judged.

The verdict: a grant keyed on a command's class is standing code execution.

The reasoning is simple once someone says it out loud. A class name doesn't bind to behavior. npm test doesn't mean "run tests" — it means "run whatever scripts.test currently says," and the agent editing your code can edit that. Grant the class, and you've granted anything that file can be made to say.

The obvious fix — hash the config files and re-ask when they change — doesn't hold either. scripts.test = node scripts/test.js moves the payload one level deeper than the hash can see. Same for tests/conftest.py, node_modules/.bin/*, and .git/config textconv drivers.

Three holes got executed against candidate designs rather than argued about:

• find . -exec inside a class called "inspect"
• git branch -D main inside a class the interface labelled read-only
• npx <tool>, which fetches and runs a package from a mutable registry

And the counterweight that killed the remaining case for it: the strict shell grammar you'd need to make classes safe refuses npx vitest run — this repo's own idiom. So the prompt-reduction payoff was small and the new attack surface was large.

What shipped instead: grants key on the exact command bytes you read in the modal, scoped to one project, surviving restarts. That inherits the session allowlist's safety property by construction — the granted string is the string that runs — while still killing the re-prompt-every-reload loop that trains people to click through without reading.

The lesson isn't "we caught a bug." It's that a roadmap row is a hypothesis, and some of them are wrong in ways only the build reveals. The row now carries a deviation note explaining what shipped instead and why, sitting next to the original spec. A backlog that only records intentions is a worse document than one that records collisions with reality.

L20 — A test suite can be green while the product lies

Here's a dialog Yee shipped:

No standing approval for this one — it uses "pnpm @dev3lop/site", which can change more than it reports. [ Deny ] [ Always allow this command ] [ Always allow here ] [ Run once ]

Read it twice. It says there is no standing approval available, directly beside an enabled button offering exactly that.

Every logic test passed. They were all correct. The button's condition was derived from one property (destructive) while the explanation was derived from another (grantable), and nothing in the suite had ever rendered the two together. A unit test can verify each half and never notice the pair contradicts itself, because the contradiction only exists on screen.

There was a second bug in the same screenshot: pnpm --filter @dev3lop/site build was refused because --filter consumes the next token, and its value was being read as the subcommand. Hence the nonsense message naming a package as if it were a command.

So we built a click-through harness. bench/app-drive.sh boots the real app hidden and evaluates scripts inside the real renderer — importing the shipped modules, opening the real modal, clicking the actual buttons, and asserting what a user would see. Zero tokens, and it runs in the chain like everything else.

87 clicks across five drivers now. The push modal, the approval modal, the endpoint settings, the secrets boundary through real IPC, the chat pills.

It earned itself on the first run by catching an unrelated regression: a refactor had collapsed rm -rf ./build into a generic "deletes files," losing the recursive-tree warning the modal used to give. No logic test would ever flag that — the string was still a valid reason.

If your interface makes claims, something has to read them.

L21 — Three of my own tests proved nothing, and mutation testing is how I know

This is the part I'd rather not write.

The discipline we adopted early: after writing a guard, seed the exact fault it claims to catch and confirm it goes red by name. Not "the suite fails" — that specific assertion, with its specific message.

It caught my own work three separate times.

The vacuous fixture. I wrote an assertion that the push modal names the branch it actually cuts from — the exact bug that had shipped and burned us. My test fixture set startBranch and defaultBranch both to main. So the assertion matched "off main" whether the code named the cut base or the branch you happened to be standing on. I reinstated the original bug and the test stayed green. It now reads the summary line with two different names.

The substring pin. I added a check that capturing the preview appends rather than replaces. It asserted a push(...) call existed in the source. I then seeded a fault that cleared the previous captures immediately before pushing — the substring still matched. Green. It reads the whole function body now.

The crash that looked like a pass. A seeded fault made a test crash mid-run. It printed no summary line, so grep FAILED found nothing and I read it as passing. Content assertions now short-circuit on error results so a mutation dies as a named red, not a silent stack trace.

Every one of those tests looked right. Two of them I'd have shipped and cited as proof.

A test that cannot fail is worse than no test, because it is counted as evidence.

The related habit that comes with it: vacuous-pass armor. Every guard sets process.exitCode = 1 and only a completed summary line flips it, plus a watchdog. Otherwise an async stall drains the event loop and exits 0 — reading as green. Suites also pin their own assertion count, so quietly losing a case is a failure rather than a smaller number nobody reads.

L22 — A number computed over lossy data is worse than no number

This one turned up three separate times in three unrelated features, which is how you know it's a real pattern and not an anecdote.

Phantom pricing. We added support for any OpenAI-compatible endpoint — OpenRouter, Groq, Together, LM Studio. The price resolver fell through to a default of $3/$15 per million tokens for anything it didn't recognise. Which meant a call to your own local server, costing nothing, would have been billed at Claude-tier rates into the spend meter, the receipts, and the generated PR body. We cannot know what a third party charges. So it's your number or an explicit unknown. Never a guess.

Silent truncation. The per-chat activity trace was capped at 400 entries with slice(-400) — which silently discards the earliest rows. Fine for a scrolling log. Fatal the moment you build an apply-success meter on it, because the rate gets computed over a biased tail and states a confidence it hasn't earned. The cap now leaves a marker recording what it dropped, and any statistic built on it reports itself as partial.

Discarded failures. Edits were only captured for training when they succeeded — the call site literally gated on success. Half the signal was going in the bin, and it was the more useful half: the instruction, the file, and the search block the model invented that matched nothing. Now failures are captured too, labelled negative at the schema level — because the row builder has a fixed field list, and a label set by the caller would have been silently dropped, putting a wrong edit into the training warehouse looking exactly like a win.

Same rule underneath all three: when you don't know, say so. An honest gap beats a confident fabrication, and it is much cheaper to fix later.

L23 — Stop treating the backlog as a queue

Halfway through, the sequencing stopped making sense.

M1 and M2 earned their place. They fixed spots where Yee was genuinely worse than table stakes — commands silently denied under concurrency, loops dead-ending with no way to continue, repo rules ignored, no way to restore a single turn. That work needed doing regardless of what Cline shipped.

Past that line, running the remaining rows in order converts into copying a better-funded team on their own axis. That's a race you lose by construction: you're always behind, by definition, and every hour spent matching them is an hour not spent on the things they can't answer.

So the grid became a menu. What gets built next is picked on one question: does this fix something that harms a real user, or widen something a competitor cannot answer?

Five rows survived that test, and they were not the next five in rank order:

Roughly the same complexity budget as one milestone. Far more value.

L24 — We are not building MCP, on purpose

Every agentic editor ships MCP. We held it, and I want to be precise about why, because "we didn't get to it" and "we decided not to" are different statements and only one of them is honest.

The case for is real, and includes a correction to my own reasoning. I argued for the compat-provider row because "one integration opens OpenRouter, Groq, LM Studio." MCP is the same shape at larger scale — one integration inherits hundreds of tool servers. I used that argument in one direction and not the other, and the difference is cost, not principle.

What's genuinely different is a cost specific to us. Yee's measured advantage is that its ecosystem — preview-awareness, redundant-command interception, a directness directive — makes frontier models spend fewer and steadier tokens for the same result. We have numbers on that. The standard MCP integration mirrors every connected server's tools into the turn-1 vocabulary and freezes it for cache stability, which means every chat pays context for every tool of every connected server, used or not. Connect three servers with a dozen tools each and you have quietly widened the exact thing you measured as your edge.

Add the rest: it's a permanent maintenance surface, and an MCP server is arbitrary code with tool access — running outside the secrets boundary we just spent a week building.

Nobody has ever chosen a code editor because it had MCP. They choose one because it does the thing they came for without lying to them.

It comes off hold the day a real user asks for a specific server by name — and when it does, the shape won't be "MCP support." It'll be the two or three servers that earn their context, enabled per-chat so a conversation that doesn't need a database never pays for one.

The backlog records this as held (⏸), deliberately distinct from not started (☐). A reader has to be able to tell an unmade decision from a made one.

L25 — Documentation rots in a specific, corrosive way

Catching the READMEs up after three weeks, I found the roadmap section listing three things as "later" that had already shipped: inline edit, @file mentions, signed macOS builds.

That's worse than merely out of date. There's an "honest list" in that README — the things Yee genuinely cannot do — and it's the most valuable section in the file. If a reader spots one item they know shipped, they stop trusting the honest list too. Stale optimism poisons stated limitations.

The honest list gained three entries in the same pass rather than shrinking: no MCP (held, with reasons), a standing approval pins the command and not its meaning, and Windows/Linux are unbuilt.

The same rot hit our own diagrams, and worse. We keep two: a system map of what the app is made of, and a test map of what protects it. Both were stale — and the test map's staleness was a bug in the tool that generates it. It cached a chain run with no way to tell the chain had changed, so every regeneration quietly re-described the previous suite. It claimed 73 steps and 34 clicks against a real 74 and 87.

That is L22 — a confident number over data that moved — living inside the tool that draws the picture of our tests. The cache is keyed to the test script now. And the system map's node count, which had been hand-typed and wrong for a week, is derived from the arrays.

Where this leaves the thesis

The first post argued that Cline's 8M developers came from trust ramps, context hygiene, and extensibility, and that Yee should learn from all three.

Building it, the trust ramps mattered most — and not because of the feature list. They mattered because every one of these lessons is the same lesson in a different costume: the system stated something that wasn't true. A modal that contradicted itself. A test that couldn't fail. A price that was invented. A trace that dropped its own history. A roadmap that listed shipped work as pending.

None of those are hard problems. They're all cheap to fix once seen. The expensive part is seeing them, and the only reliable way we found is to make the software prove its claims to something that doesn't take its word for it — a mutation, a click, a marker recording what got dropped.

That's what the remaining 30 rows are being judged against now. Not "does Cline have it."

Yee is a local-first AI code editor. The 81-row grid, the 74-step test chain, and both maps live in the repo — the test map is a standalone page you can open with no server: every step, what it protects in plain English, and the incident that caused it to exist.