The cache trap: when a system fails identically every time

Two runs of the same prompt produced byte-identical test files — both containing a test that asserted a plus sign would lowercase to False. The system wasn't broken randomly. It was broken reproducibly, which is worse.

Two runs of the exact same prompt produced byte-identical test files. Not similar — identical, down to the byte count. And both contained the same nonsensical test, one that asserted a Python string's .lower() would return False for the character +. Which is impossible. A plus sign has no case.

The system wasn't broken in a random way. It was broken in a perfectly reproducible way. Yun ang nakakatakot (that's the scary part).

What was actually happening

The LLM client had a cache. Reasonable idea: memoize completions so a repeated call doesn't burn a repeated API bill. It keyed on the provider, the prompt, the system message, and the model.

So one day a model rolled badly on a calculator prompt and produced an impossible test. That bad output got cached. And every subsequent calculator run inherited it — because the code-generation stage was deterministic enough to produce near-identical source, which produced identical tester prompts, which hit the cache, which replayed the same garbage. Forever.

This is worse than a flaky system. A flaky system tells you something's wrong. A system that fails identically every time looks like a real bug in your code, and you'll spend days hunting a phantom in the generator while the actual culprit sits in a dictionary.

The fix isn't "tune the cache"

My first instinct was cache-size, TTL, eviction policy. All wrong. Those just delay the trap.

The real fix is recognizing that caching is a per-call policy decision, not a global setting. Some LLM calls are genuinely idempotent — a router deciding "is this request codegen or debugging?" will always give the same answer for the same input, and caching it is free money. But generative work is the opposite: variance across runs is the entire point. If code generation returns the same thing every time, retrying it is pointless.

So every agent now declares whether its calls are cacheable. The router: yes. Everything that generates — code, tests, architecture, reviews, requirements, debugging: no.

The bonus bug

While plumbing the cache policy through, I found that a config helper was silently dropping the fallback provider and fallback model fields. Which meant the entire failover system — the thing that's supposed to switch to a backup model when the primary is down — had been a structural no-op since the day it shipped.

It had never once worked. Nobody noticed, because the primary had never gone down during a test. That's the kind of bug that waits patiently for your worst day.

The week's real work was strategic

The code changes above were small and surgical. The bigger move was a set of decisions about what this project is, and what it deliberately isn't.

I split the vision into two committed products and one deferred one:

• The greenfield SDLC platform — what this repo is. Generates new code, tests, design, and an audit trail from a prompt. For regulated teams starting new projects under contract.
• A read-only knowledge platform — a separate future repo. Local-first retrieval over enterprise knowledge scattered across issue trackers, wikis, code hosts, and QA assets. The one-liner I landed on: a greenfield tool built specifically to make brownfield work tolerable.
• A write-enabled sibling — deliberately deferred, design captured, not built.

The architectural decision I'm proudest of

The knowledge platform has one hard rule: read-only by design. Every connector refuses to write. Not "write with a permission flag" — structurally incapable of writing.

The reasoning is a risk-class argument, not a feature argument. The worst case for a read-only tool is "it gave me a wrong answer." The worst case for a write-enabled one is "it silently modified production." Those are not the same category of bad. Regulated buyers can accept the first and cannot accept the second.

I could have made the connectors read-write with a permission system. Instead I made writes impossible. The second version is shorter, safer, and easier to sell.

Why defer the third product

Honest competitive reasoning: brownfield code modification is the most crowded of the three categories. Every major AI coding tool lives there. The wedge for a differentiated entry is real but narrow — audit-defensibility at the top of the regulated curve, where existing tools genuinely fail.

A speculative build into a saturated market, with a small team, is high-risk and expensive. Deferring costs nothing: the design is captured, the shared-library plan supports it later, and no current decision is blocked. If a customer shows up with budget and a pilot commitment, the activation criteria flip and the build is half the size it would be today.

The general principle: starting separate can be unified later. Starting unified can never be cleanly split. Two products keeps the option to add a third open, without spending engineering effort to hold it open.

And why any of this is written down

Most of this week's work was strategic. Strategic work that lives only in a chat window is a year-old TODO nobody remembers deciding.

def test_returns_false_when_operation_is_lowercase_plus():
    assert "+".lower() is False   # impossible: "+" has no case