Most "AI flakiness" was infrastructure flakiness

I spent days convinced the model was the problem. Random hangs, empty error messages, runs dying mid-pipeline. It turned out to be a file watcher, a timeout, and an event loop policy. The models were fine the whole time.

I spent days convinced the AI was the problem. Every run of the multi-agent pipeline would fail somewhere different — a hang here, an empty error message there, a mysterious mid-run death. Classic "AI is flaky" territory. The kind of thing you shrug at and add a retry to.

It wasn't the AI. Almost none of it was the AI.

Three bugs wearing an AI costume

The dev server's file watcher was watching the artifacts directory — the same directory the pipeline writes to. Every time an agent produced output, the server restarted itself mid-run. Random deaths, no pattern, no error. The pipeline was being assassinated by its own tooling.

A 60-second timeout on model calls was firing on slower requests, which triggered the retry library, which meant the call was silently re-issued while the original was still in flight. From the outside: a hang. From the logs: nothing useful.

And on Windows, the async event loop policy made subprocess calls fail with empty error messages. Not a bad error. No error. Just a failure with nothing attached to it.

Why this matters more than it sounds

Nagpakita ito ng isang bagay (this revealed something) about debugging systems with a probabilistic component in them: when part of your system is allowed to be non-deterministic, every other bug gets to hide behind it.

A file watcher killing your process is a boring, findable bug in a normal system — you'd have it in an hour. Put an LLM in the same pipeline and suddenly the same symptom reads as "the model hung," "the model returned garbage," "the model is being weird today." The non-determinism becomes a universal alibi.

So the discipline that came out of this: when something fails in an AI pipeline, assume it's infrastructure until you've proven it's the model. Reverse the usual instinct. The model is the last suspect, not the first.

That inversion is uncomfortable, because blaming the model is the path of least resistance. Prompt tweaks feel like progress. But a prompt tweak that appears to fix a watcher bug is the worst possible outcome — you've now got a superstition baked into your system, and the real bug is still there waiting.

What "fixed" actually looked like

Once the platform could complete a run without being murdered by its own tooling, the first real end-to-end pipeline landed: a prompt in, and out the other side came generated code, six passing tests, a design document, and an audit trail. All from a chain of agents each doing one job.

That first clean run was the baseline everything since has been measured against. And it only existed because I stopped tuning prompts and started reading server logs.