We rebuilt our agent suite into a product team. Here's what the evidence said.

J, the agent suite my brother and I built, was good at exactly one thing: taking a written ticket to a reviewed pull request. That is a real capability and it shipped real code. It also has an obvious ceiling — somebody still has to write the ticket, and writing the ticket is most of the thinking.

So we rebuilt it to cover the whole lifecycle — designed so that a sentence describing a need goes in and a deployed product with tests and monitoring comes out. This is what we learned building it, including the parts that were unflattering.

We read the evidence before writing anything

The intuitive design for “an AI dev team” is to simulate a company. Give agents job titles, let them hold meetings, let a CEO agent arbitrate. That generation of frameworks exists — MetaGPT, ChatDev — and it did not survive contact with real repositories. Both pivoted or froze.

Every system that actually ships in 2026 is one of three shapes: a single strong agent working single-threaded, parallel agents on genuinely independent tasks in isolated checkouts, or one writer plus specialists whose job is verification rather than production. Under matched compute budgets, a strong single agent equals or beats role-play multi-agent — most reported gains from “more agents” turn out to be the extra tokens, not the architecture.

The research is blunter still on the failure modes. Across sixteen hundred annotated traces of multi-agent systems, the failures cluster in system design, not model capability: unclear specification, agents misaligned with each other, and weak verification. And letting agents debate toward consensus actively hurts — accuracy can drop over debate rounds as models flip from correct to incorrect under peer pressure.

So we did not build a company. We built a spine of artifacts and gave a small number of roles very sharp boundaries.

The abstraction that mattered was not about agents

The single highest-leverage change had nothing to do with AI. J was wired to Jira at every level — fetch the ticket, comment on the ticket, transition the ticket — which meant a weekend project needed a Jira instance to use any of it.

We introduced a work item abstraction with two backends: a Jira issue, or a task entry in a markdown file inside the repository. Every capability now calls fetch, comment, transition — never a provider directly. The Jira path is untouched for client work; personal projects run the same pipeline against a text file.

It took a sweep of twenty-five call sites and it unlocked more than any new capability would have. The lesson generalises: when a system is hard to reuse, the blocker is usually a dependency nobody noticed was load-bearing.

Verification you cannot fake alone

If I had to keep one piece of the suite and throw away the rest, it would be this: one reviewer on every change is blind by design.

It receives the diff and nothing else. No plan, no ticket, no summary of what the author was trying to do. Every other reviewer gets that context, and context makes them read the code the way its author meant it. You cannot un-know your own intent — which means a fresh, uninformed reader is not a prompting technique. It is an architectural property you either have or do not.

The second half is refutation. A reviewer asked to find problems will find some, whether or not they exist, and chasing every finding is how a review turns into over-engineering. So every substantive finding faces agents whose instruction is to demonstrate it is not real, defaulting to dismissal when they cannot. Only what survives reaches me.

Measure the binary, not the documentation

We wanted to replace our hand-rolled agent queue with the runtime’s native background agents. Before writing anything, we checked what the installed version actually supported.

The command help listed no session-management commands, which read like a definitive no. A control test against a deliberately invalid command showed otherwise: the real commands exist and print their own usage, they are just hidden from the listing. Had we trusted the help output we would have concluded the wrong thing — and had we trusted the documentation instead, we would have concluded the opposite wrong thing, because the docs describe capabilities the installed binary does not expose.

We dispatched a probe agent and exercised every operation. The verdict: four of six work, and the two missing ones are the two our epic flow depends on. So we did not migrate. That is a better outcome than either guess, and it cost twenty minutes.

Machinery decays, quietly, in exactly the ways it exists to prevent

The most uncomfortable finding came from auditing our own system.

The blind reviewer was not blind. One skill carried a local copy of the reviewer configuration, frozen months earlier, that handed it the full contents of every modified file. The safeguard I have just spent several paragraphs praising had been silently broken since April, in the one skill whose entire job is review.

It gets worse. Seven of sixteen skill bundles were missing the rule that keeps a test author from editing the code under test. And the reason nobody had fixed any of it was a governance document declaring all those stale copies “intentional — must not be deleted.” Eight of its ten claims were factually false. Nothing in the build even read the file. It existed purely to tell reviewers not to touch the broken parts.

We deleted it and replaced it with a check that fails the build when a local copy diverges from the canonical one — then proved the check works by introducing a drift on purpose and watching it fire. Governance that a machine enforces beats governance that a document asserts.

Tests beat reading, every time

The suite is specified, evaluated against two hundred and seventy test cases, and it builds clean. That last part matters more than it sounds.

Three separate defects in our own system were caught by tests rather than by reading it — including the blind reviewer that had stopped being blind. Reading is what I would otherwise have relied on, and reading found none of them. The gap between what a document says and what its author believes it says is invisible from the inside, which is the entire argument for making a machine check it.

So the discipline we ended up with is the one we now apply to everything: write the standard down once, make something mechanical enforce it, and treat “I read it and it looked right” as the weakest possible evidence.

Next up is taking a small project through the whole chain, start to finish, and measuring what it actually costs — tokens, wall-clock, and how many times it needs me. That post will have numbers in it.

The case study on J covers the current architecture, and how I build has the operating model behind it.