The router that solved itself out of a jobproject

A few posts back, my homelab’s GPU got its first real AI tenant: a small local model auditioning as the router at the front of my alert pipeline, deciding which alerts deserve a full cloud-model investigation and which are noise. The deal I struck with it was explicit. It would run in shadow, recording the call it would have made next to what the trusted pipeline actually did, and it would take over only when the record proved it out. The prize was economics: at its peak, the shadow log showed it would have skipped 58% of my cloud investigations, thousands of small decisions a year moved onto hardware I own.

The audition went well. The router graduated from the original 9-billion-parameter pick to the larger 14-billion-parameter model the rest of the lab now shares, and when a battery of injected fault drills exposed a blind spot for faults that look benign on the surface, the fix was switching the model’s reasoning mode on and accepting five seconds of thinking time. By graduation day it was, on quality, everything I’d asked for: it had caught every one of the 8 real faults in its final scoring window, nothing it would have skipped proved real, and the 10 times its own model server was unreachable mid-upgrade, the pipeline failed open and sent the alert onward instead of dropping it. Every safety bar, cleared.

Then I read the one number the project actually existed for. The would-skip rate, 58% when I wrote the shadow’s job description, had collapsed to 11%: of the 53 alerts the shadow watched in its final week, it would have spared the cloud exactly 6 investigations. Flipping it live would have saved less than one cloud call a day. My cost-cutting router had, over its months in shadow, become free labor for a bill that no longer existed.

The root cause is the good kind of embarrassing: a different project ate this one’s lunch. The 58% was measured against the old firehose, around 18 investigations a day, roughly 90% of them benign churn (routine logins, backup jobs, a chatty photo importer). Over the same weeks the router sat in shadow, I had been tuning that noise out at the source: excluding known-benign patterns upstream, fixing a log-shipping bug, teaching the alert rules what my lab’s normal actually looks like. The firehose shrank to 5-8 investigations a day, and what still gets through is mostly crash-shaped or genuinely ambiguous, exactly the traffic a competent router should escalate anyway. The job I had hired it for was turning away the flood, and I had already drained the flood myself.

I’ll admit I reached for reasons to keep it anyway. The best-sounding one was “keep the shadow running for training data,” and it fell apart on inspection: the evaluation corpus accrues from the pipeline’s own investigation records whether or not a shadow rides along. The revisit trigger doesn’t need it either; if investigations ever creep back past around 25 a day with the benign share climbing, that regrowth is visible in the same dashboards that caught this collapse. Retirement won. One config line turned the shadow off, and the essay I’d just written about scaffolding melting got its first live demonstration a week later.

The goal, cutting cloud spend on benign churn, was achieved beyond the original target; the router just wasn’t the component that achieved it. And the durable yield was never the on/off switch: it was the evaluation machinery built to decide whether to trust the thing. A corpus of some 1,300 real alerts plus 25 injected drills, a replay harness, and a hardened prompt now sit ready to score any future model release against my actual traffic in an afternoon. The router is retired; the exam it forced me to write is permanent. If the noise ever regrows, re-arming is one config line and two weeks of fresh shadow evidence, and this time the exam is already written.