A cancel button that lies, and retrieval that returns garbage confidently
June 29 – July 10, 2026Build LogDebuggingRAGPatterns
Two problems six weeks apart that turned out to be the same problem. A cancel button that stopped the UI but not the work. A search that confidently returned a table-of-contents fragment. Both fixed by refusing to make the primary mechanism smarter.
Two problems, six weeks apart, that turned out to be the same problem.
Problem one: a cancel button that lies
A debugging run had been spinning for 219 seconds. There was a Cancel button under the spinner. I clicked it. The UI said "cancelled."
The backend kept running.
The first implementation was the easy half: an abort controller on the frontend, a cancel button wired to it, the fetch promise rejecting, the UI swapping to a cancelled state. From the user's seat that looks complete. It isn't. The server-side task was still awaiting the model call. Tokens kept burning. The audit log still recorded a successful completion for a request no client was listening to. Restart the server and you'd see the work finish in the logs, minutes later, for a run the user believed they'd stopped.
A cancel button that doesn't propagate to the real work is worse than no cancel button. It teaches the user to trust a control that's lying to them.
The real fix had three parts, all shipping together: the in-flight work has to be tracked as a handle the cancel path can reach; there has to be an endpoint that actually cancels it; and when cancellation propagates, the audit log has to record that — not a phantom success.
The frontend then does two things in order: tell the backend to cancel first, then abort the local request. Backend-first is what frees the model call and writes the truthful event. The local abort is best-effort UI cleanup. If the cancel call fails — network down, backend gone — the abort still fires so the user isn't stranded staring at a spinner, but the backend task continues. That's a known degraded mode. Known, and stated, rather than silent.
The caveat I'm being honest about
Cancellation fires immediately in the async task. But when the model runs as a subprocess, that subprocess may keep going for several seconds while the OS cleans it up. The HTTP contract honors the cancel instantly and the audit event fires on time — but full transport-layer teardown needs a different subprocess API with a handle the cancel path can terminate. That's queued as its own piece of work, hindi ko binabalewala (I'm not waving it away).
Problem two: retrieval that returns garbage confidently
Six weeks later, a different system. The knowledge search UI, asked "what is SysML", returned three results. The top hit was a table-of-contents fragment — a line of dots and a page number. The second was German-language security prose. The third was a chapter about dependency injection in C#.
All three scored around 0.20. Near-random.
The tempting fixes were all single-layer: tune the embedding model, swap the chunker, add a reranker. But the failure was compounded — three separate things wrong at once, and fixing any one would have produced a slightly-less-bad version of the same garbage.
The deepest of the three: the chunker cuts documents into fixed-size character windows. A page containing 9.4.16.1 Vector Functions Overview.........395 gets indexed as a legitimate chunk. Its embedding lands near "what is sysml" because the word vector and a numeric neighborhood happen to overlap. Nothing about the retriever is broken. The input to it is garbage.
The pattern, stated once
Retrieval scoring alone will never guarantee chunk quality. The response isn't "tune the embedding model harder." It's "gate the chunks that carry no information at all, before they can be retrieved or shown to a model."
Two gates, same rules. One at ingest, so bad chunks never enter the index. One at answer time, so a legacy bad chunk sitting in an older index can't reach the model anyway. Defense in depth — because retrieval quality is a moving target, and any single gate eventually gets circumvented by a corpus change or a chunker tweak.
Four fixes, one shape
Once I saw it, the pattern was everywhere in this project:
The cache trap — the fix was a per-call cacheability policy, not "tune the cache."
The reviewer abstain bug — a three-tier decision resolver with a prose fallback, not "make the model output better JSON."
The cancel button — backend task tracking plus real cancellation propagation, not just a UI abort.
The retrieval garbage — quality gates at ingest and at answer time, not a better embedding model.
Every single one was multi-layer defense chosen over a single-layer "make the primary mechanism smarter" instinct.
The primary mechanism is never the place to put the guarantee. The cache, the reviewer, the abort controller, the retriever — each is a mechanism doing its job correctly. The guarantee has to live somewhere the mechanism can't fail you.
Why I keep relearning this
Because the single-layer fix is always the one that feels like engineering. Tuning the embedding model is interesting work. Writing a quality gate that throws away chunks made of dots and page numbers is boring plumbing.
The boring plumbing is the thing that holds. Matagal pero matibay (slow but solid).
The cancellation contract and the retrieval rebuild
The implementation detail, the code, and the specific tooling. Same password as the other build-log posts — or request access below.
Honest cancellation, in three pieces
1. Track the in-flight dispatch as a task
A module-level dict _inflight_fix_tasks: dict[str, asyncio.Task] keyed by run id. The endpoint wraps the agent dispatch in asyncio.create_task(...), stores it in the dict before the await, and pops it in a finally regardless of how the await exits.
Storing-before-awaiting matters: it prevents a race where the cancel endpoint runs between task creation and dict insertion.
2. A cancel endpoint that actually cancels
POST /api/runs/{run_id}/fix-with-ai/cancel looks up the task, calls task.cancel(), returns 202. Returns 404 if no task is in flight for that run — a cleaner contract than 200 + "nothing to do," because the client should know whether its cancel landed.
3. CancelledError handling that writes audit truth
When cancellation propagates through the await, the endpoint catches asyncio.CancelledError, emits an agent-failed event with reason=cancelled at WARNING severity (not ERROR — a user-initiated stop is not a bug), then surfaces HTTP 499 so the client knows the backend honored the cancel rather than silently completing.
A 409 guard on the dispatch endpoint catches the "fire two fixes on the same row" race before the second one gets a task. The UI already disables both buttons while a fix is in flight — but correctness shouldn't rely on UI invariants.
The subprocess caveat, precisely
asyncio.CancelledError fires immediately in the task. But when the provider runs the model call as a subprocess inside asyncio.to_thread + subprocess.run, the subprocess itself may keep running for several seconds before kernel-level cleanup completes. Full transport-layer teardown requires switching to subprocess.Popen with a process handle the cancel path can call terminate() on. Queued as its own follow-up.
The retrieval rebuild
The screenshot said three things at once:
The UI returns raw chunks. The search endpoint is a retrieval debugger — there is no model step. What I was staring at was the recall pool, not an answer. Real answers need a Q&A endpoint that puts a model between retrieval and the user.
The chunker is a character window. Fixed 1000-character slices. A ToC page becomes a "legitimate" chunk.
The corpus was missing the answer entirely. Roughly 50 books were never ingested — a combination of a parser failure and a virtual-environment issue. The retriever was doing its best over a corpus that didn't contain what was being asked for.
What the rebuilt pipeline looks like
retrieve → quality-filter → synthesize with citations → refuse honestly when there are no strong sources.
That last clause is the one I care about. When the quality gate leaves nothing above threshold, the answer is "I don't have good sources for this," not a confident synthesis over three bad chunks. A retrieval system that can't say "I don't know" will lie to you at exactly the moments it matters.
The quality gate
Same rules applied twice. At ingest, so bad chunks never enter the index. At answer time, so a bad chunk from an older index build can't reach the model. Chunks that are predominantly punctuation, page numbers, or navigational scaffolding carry no information and get dropped regardless of what they score.
Context engineering over prompt engineering
The strategic doc that came out of this argues a position worth stating plainly: for local models doing judgment work, what you put in the context window matters more than how you phrase the instruction.
A large cloud model can often recover from a mediocre context — it has enough general knowledge to fill gaps. A small local model cannot. It will faithfully synthesize whatever you hand it, including three chunks of noise. Improving the prompt does nothing. Improving the context does everything.
This reframes a lot of "the local model isn't good enough" complaints as "the retrieval isn't good enough," which is a much more tractable problem.
Files touched (cancellation)
The runs API route — the in-flight task dict, the rewritten handler body with try/except CancelledError and finally pop, and the new cancel endpoint.
The runs frontend module — the cancel handler now POSTs backend cancel first, then aborts the local controller.
The runs stylesheet — cancel button styling, sized for the spinner row. (Shipped earlier in the day with the frontend-only cut, which is exactly the half that didn't work.)
Build log entry from an append-only debugging journal. The blog is the narrative; the session docs are the audit trail.