The failure mode wasn't the LLM — it was the contracts between agents
May 14–17, 2026Build LogArchitectureAgents
Once infrastructure stopped killing runs, the real bugs surfaced. Codegen wrote source and the tester independently guessed what was in it. The reviewer judged code it couldn't see. Every implicit assumption between agents was a place a future bug could hide.
Once the infrastructure stopped killing runs, the real failures showed up. And they weren't where I expected.
The pipeline is a chain of agents: one writes requirements, one designs architecture, one generates code, one writes tests, one builds and runs them, one reviews the result. Each is an LLM call with a job. The obvious failure mode is "the model produced bad output." Hindi yun ang nangyari (that's not what happened).
The pipeline's failure mode wasn't the LLM. It was the contracts between agents.
Three ways an implicit assumption becomes a bug
Codegen wrote source. The tester independently guessed what was in it. The tester read the generated file and inferred which symbols to test by parsing it. Two agents, same source, two independent interpretations of what the module exports. Any disagreement was a silent bug.
The reviewer judged code it couldn't fully see. It got the source. It did not get the tests, nor the verdict from the stage that had just run them. So it would confidently announce "no tests present" moments after the build agent had executed the test suite successfully. The reviewer wasn't hallucinating — it was answering honestly about a context I'd failed to give it.
Retry loops burned tokens on identical errors. An attempt would fail with an import error; the retry would get the exact same generic "please fix this" prompt; it would produce the same code; it would fail with the same error. Three attempts, sixty seconds each, zero information gained.
The fix is the same shape every time
Turn an implicit assumption into explicit data flow.
Codegen now publishes its export list as structured output, and the pipeline forwards that list to the tester. The tester binds to the explicit contract instead of re-deriving it. One entire class of drift, eliminated — not by making either agent smarter, but by making the handoff visible.
The reviewer's payload now carries the tests and the build verdict. Same model, same prompt structure, dramatically better reviews — because it can finally see the thing it's being asked to judge.
And retries got two upgrades: a fingerprint of the error so two identical failures in a row break the loop early instead of burning a third attempt, and error classification so the retry prompt is targeted. An import error gets "make every public name top-level." An assertion failure gets "the test is the spec — don't weaken it." Different failure, different advice.
The garbage-cascade incident
This one taught me the most.
A CLI subprocess failed but returned exit code zero (a Windows wrapper quirk). Its stdout contained a mangled concatenation of an error message and a batch-file prompt: Execution errorTerminate batch job (Y/N)?
The requirements agent's fallback path dutifully stored that string as the project's requirement. The architect agent then dutifully designed an architecture to satisfy it. Two more stages built on top. Four LLM calls, each doing exactly what it was told, each building on nonsense.
Never let a fallback silently continue the pipeline. Fallbacks are placeholders by design. Treat one as a real output and every downstream agent builds on garbage — and the human reviewing it has to unwind four stages of bad context to understand why the run produced nothing.
The fix: detect the fallback at every critical agent boundary, halt cleanly, and hand the user a structured set of choices — try again, wait, fix the model connection and resume, or cancel. Halt loud, halt early, give the human buttons.
Then a clean run — and what it revealed
Eventually the pipeline produced its first fully clean end-to-end run. No human gate, artifacts on disk, green all the way through. Sarap sa pakiramdam (felt good).
Then I pointed it at a harder prompt and got back a 22-line file that implemented one enum out of the four things the prompt asked for. The reviewer's verdict: "Correct and well-tested. Ready to merge."
Three stacked failures, each feeding the next. But the deepest one is a lesson I'd underestimated:
"Approved" doesn't mean "complete." The reviewer was judging whether the code that exists is consistent with the tests for it. It never read the requirements. So a tiny, internally-consistent implementation reads as perfect — because by the only standard it was given, it is perfect.
That's not an AI bug. That's me giving a reviewer the wrong job description and then being surprised it did that job well.
The discipline that came out of it
Mid-session, watching myself layer a fifth fix on top of four unverified ones, I said out loud: stop shipping new code until you have a verified baseline.
Five fixes with zero confirmed clean runs between them means five unknowns. The disciplined move was to stop, fire a simple prompt, watch the events live, and let the data diagnose. Once a green run landed, the next three bugs each had a clear scope — because I finally had a known-good state to compare against.
The contracts, the abstain bug, and what shipped
The implementation detail, the code, and the specific tooling. Same password as the other build-log posts — or request access below.
The symbol contract, concretely
Codegen publishes its export list — top-level public classes, functions, and ALL_CAPS constants — in its structured result under an exports key. The pipeline forwards it to the tester as subject_symbols. The tester binds to that explicit list, falling back to local AST inference only for callers that don't supply one.
Same idea for the reviewer: the pipeline packs tests and build_summary into the reviewer's artifact dictionary, and the reviewer prompt surfaces them as labeled blocks. That single change killed the "no tests present" hallucination outright.
Convergence detection and error routing
A _fingerprint_build_error(findings) helper hashes the shape of a failure. Two consecutive attempts with the same fingerprint → break the codegen loop, emit AWAITING_HUMAN with reason: convergence_detected. No more burning three identical attempts.
Alongside it, _classify_build_error(findings) returns one of import | module | assertion | syntax | timeout | other. A lookup table maps each class to a fix-strategy block that gets prepended to the retry prompt. Targeted prompt, targeted fix.
The reviewer abstain bug — three layers, and which one mattered
Every run was hitting a human gate because the reviewer kept returning decision: abstain while clearly approving in prose. The cause was one concrete thing: the model (a smaller, faster one used for review) ignored the "return only JSON" instruction and emitted markdown with verdicts like "✅ Correct, well-tested, ready to merge." The parser found no JSON, defaulted to abstain, and abstain wasn't approval, so it retried, exhausted attempts, and escalated.
Three layers shipped:
Tightened the system prompt — JSON-only instruction moved to the end, explicit forbid of markdown, one-shot examples.
A prose-fallback parser — when JSON extraction returns nothing, scan the prose for unambiguous verdict phrases ("ready to merge", "verdict: reject", "do not merge"). Approve or reject only on clear signals; ambiguous prose stays abstain.
Abstain halts immediately — no retries. The same artifact and the same prompt will not make a model more decisive. Escalate to a human on attempt one rather than burning two more calls to reach the same non-conclusion.
The clean run proved which layer carries the weight: layer 2. The model still emitted markdown after the prompt rewrite. The prose-fallback caught "the implementation is correct" and inferred approval.
The safety net you build for "the model should have followed instructions" ends up being the thing that actually works. Not the better prompt. The net.
The multi-block code extraction chain
Three stacked bugs in the 22-line-file incident:
The code extractor took only the first fenced block. The model produced one block per class; blocks two onward were silently dropped. Fixed by concatenating all blocks.
The tester writes tests for the source, not for the requirements. When codegen omits scope, the tester writes tests for what's there, not what should have been there. Queued.
The reviewer judges internal consistency, not spec compliance. It never reads the requirements or design artifacts. Queued.
And the fix to #1 immediately opened a new failure: naive concatenation produced a file that imports from itself, because the model's multi-block response wasn't "several classes, one file" — it was "several files." Line 70 of the concatenated output imported a module that didn't exist, in a file that was itself meant to be that module.
Also shipped in this stretch
Per-stage manual rerun. An endpoint that dispatches exactly one agent using on-disk checkpoints, under the parent run. The UI exposes it as a dropdown on every terminal row, so a human can override the automatic loop's choice.
Scoped the injection scanner to the user boundary. The heuristic prompt-injection scanner was running on every LLM call, including agent-to-agent prompts containing legitimate technical text like "developer mode" as a feature description. It was blocking codegen retries by flagging codegen's own prior output. Fix: internal calls trust the chain; only the agent receiving raw user input opts into scanning.
Conditional requirements style. Formal "the system shall…" phrasing now triggers only on keywords indicating a government or contract context. Everyday prompts get plain English. Same agent, different system prompt per request.
A "what to type" guidelines panel with click-to-insert examples, making the platform self-documenting.
Two design docs — a formal pipeline state machine (six run-level states, six stage-level states, a transition table for every endpoint, and six identified gaps in the current implementation), and a target architecture in seven layers. Written because the pipeline had accumulated nine overlapping endpoints and six distinct pause sites with ad-hoc rationales. Rather than patch a tenth, write down what the thing is supposed to be.
The headline that generalizes
Most "the pipeline isn't working" complaints turned out to be configuration or contract drift, not code bugs. A CLI provider's coding-agent defaults leaking through a print mode — fixed by routing that stage to a different model, no code change. The scanner false-positiving — one default-flag change. Formal phrasing on every prompt — a conditional system prompt.
Each of those looked like a multi-day "fix the agent" rabbit hole. Each was a few-line configuration change once the right boundary was identified. Finding the boundary is the work.
Build log entry from an append-only debugging journal. The blog is the narrative; the session docs are the audit trail.