I spent days convinced the model was the problem. Random hangs, empty error messages, runs dying mid-pipeline. It turned out to be a file watcher, a timeout, and an event loop policy. The models were fine the whole time.
I spent days convinced the AI was the problem. Every run of the multi-agent pipeline would fail somewhere different — a hang here, an empty error message there, a mysterious mid-run death. Classic "AI is flaky" territory. The kind of thing you shrug at and add a retry to.
It wasn't the AI. Almost none of it was the AI.
Three bugs wearing an AI costume
The dev server's file watcher was watching the artifacts directory — the same directory the pipeline writes to. Every time an agent produced output, the server restarted itself mid-run. Random deaths, no pattern, no error. The pipeline was being assassinated by its own tooling.
A 60-second timeout on model calls was firing on slower requests, which triggered the retry library, which meant the call was silently re-issued while the original was still in flight. From the outside: a hang. From the logs: nothing useful.
And on Windows, the async event loop policy made subprocess calls fail with empty error messages. Not a bad error. No error. Just a failure with nothing attached to it.
Once those three were fixed, the AI behavior was — and is — actually pretty good. The models were doing their job the whole time. I just couldn't see it through the noise my own infrastructure was making.
Why this matters more than it sounds
Nagpakita ito ng isang bagay (this revealed something) about debugging systems with a probabilistic component in them: when part of your system is allowed to be non-deterministic, every other bug gets to hide behind it.
A file watcher killing your process is a boring, findable bug in a normal system — you'd have it in an hour. Put an LLM in the same pipeline and suddenly the same symptom reads as "the model hung," "the model returned garbage," "the model is being weird today." The non-determinism becomes a universal alibi.
So the discipline that came out of this: when something fails in an AI pipeline, assume it's infrastructure until you've proven it's the model. Reverse the usual instinct. The model is the last suspect, not the first.
That inversion is uncomfortable, because blaming the model is the path of least resistance. Prompt tweaks feel like progress. But a prompt tweak that appears to fix a watcher bug is the worst possible outcome — you've now got a superstition baked into your system, and the real bug is still there waiting.
What "fixed" actually looked like
Once the platform could complete a run without being murdered by its own tooling, the first real end-to-end pipeline landed: a prompt in, and out the other side came generated code, six passing tests, a design document, and an audit trail. All from a chain of agents each doing one job.
That first clean run was the baseline everything since has been measured against. And it only existed because I stopped tuning prompts and started reading server logs.
Headline lesson: Most "AI flakiness" was infrastructure flakiness. The models were fine. My watcher, my timeouts, and my event loop were not.
The three bugs, and what shipped alongside
The implementation detail, the code, and the specific tooling. Same password as the other build-log posts — or request access below.
The three infrastructure bugs, concretely
1. The reload watcher
The uvicorn dev server was running with --reload and its watch path included artifacts/ — the directory every agent writes its output to. Agent finishes a stage, writes an artifact, server sees a file change, server restarts. The multi-stage run dies at a random point depending on which agent happened to finish first.
Fix: exclude the artifacts directory from the reload watcher. One line of config. Days of confusion.
2. The timeout / retry interaction
A 60-second LLM timeout was tripping on slower completions. The retry decorator (tenacity) caught the timeout and re-issued the call — while the original request was often still running server-side. The user-visible symptom was a hang; the logs showed a retry with no indication the first call was still alive.
Fix: raise the timeout to something realistic for the model and prompt size, and make sure the retry policy doesn't stack on top of an in-flight request.
3. The Windows asyncio subprocess failure
On Windows, the default event loop policy in some contexts is the Selector loop, which does not support subprocess creation. The failure surfaced as a subprocess call returning an empty error — no exception message, no stderr, nothing.
Fix: run the subprocess via asyncio.to_thread + subprocess.run rather than the native async subprocess API. Works on any event loop policy. This became the pattern for every CLI-based provider in the system.
What shipped alongside
Security audit + dependency fixes — swapped a JWT library for a better-maintained one, bumped crypto and multipart floors, added a lockfile with hashes, a SECURITY.md, and a CI supply-chain audit job running against the OSV vulnerability database.
A CLI provider integration — with the Windows PATHEXT resolution problem solved via shutil.which, plus the asyncio subprocess fix above so it works regardless of event loop.
A persona registry — agents no longer hardcode a single persona. Personas resolve per-agent from config, so a "senior coder" agent and a "strict reviewer" agent can carry genuinely different system prompts.
Self-build defenses in every code-producing agent. Codegen, tester, and debugger each run their own output through compile() plus an import check before handing off. If it fails, they retry inline with the real error fed back to the model. Nothing leaves an agent unless it at least parses and imports.
AST-level tester defenses. A function walks the generated test file's AST and strips any top-level redeclaration of a symbol the subject module already defines. That's a mechanical guarantee — it doesn't depend on the model following an instruction. Prompts ask; ASTs enforce.
A pipeline stepper UI — seven stages, status icons, click through to inline detail per stage.
The pattern that keeps recurring
That last item — the AST stripping — is worth pulling out, because it's the first appearance of a shape that shows up in every later session:
Prompts ask. Code enforces. Anywhere a guarantee actually matters, you can't rely on the model following an instruction — you build a mechanism that makes the bad output structurally impossible or catches it before it propagates. The prompt is a hint. The AST walk is a contract.
I did not know at the time how often I'd end up relearning that.
Build log entry from an append-only debugging journal. The blog is the narrative; the session docs are the audit trail.