The homelab grows a brain of its ownproject
The new graphics card in my homelab is handy for transcoding video, but that was never its true purpose. For months I’d been renting every small AI decision in the lab from a cloud API, lately Claude Sonnet 5. Not the big open-ended work; the little bounded judgments, like whether a given log line is worth waking me up. Each call is cheap; thousands of them are not, and every one ships a piece of my house to someone else’s server and makes my alerting depend on their uptime. I wanted the small, constant decisions to happen at home, on hardware I own, like most every other digital service in my life.
So the GPU got a second job: a local model server (an Ollama endpoint the rest of the lab can call, exactly like the cloud API it’s replacing). The first customer picked itself. My alert pipeline has a router at the front, a model that reads an incoming alert and decides whether it’s worth a full investigation or should be discarded as just noise. That runs thousands of times a week, it’s bounded and well-specified, and it was going to the cloud. The only real question was whether a model small enough to run at home was good enough to trust with it.
I didn’t want to answer that by vibes, so I ran a bake-off on 65 real alerts pulled from the pipeline’s own history, each labeled with what it actually was: a real problem, or noise. That known outcome was the answer key. Six models sat the exam, five of them local and one the cloud incumbent (Claude Opus 4.8, the frontier flagship at the time) as the baseline to beat. The bar was safety first: any model that ever waved a real fault through as noise was disqualified, however sharp it looked otherwise. One local model did exactly that, filtered noise beautifully, and missed a genuine problem once. Out.
Here are the results in full:
| Model | Where it runs | Caught every real fault? | Benign correctly called benign | Cloud investigations it would skip | Mean decision time |
|---|---|---|---|---|---|
| Qwen 3.5 9B | Local (RTX 3090) | Yes (7/7) | 50% | 45% | 1.4 s |
| Qwen 3.6 27B | Local (RTX 3090) | Yes (7/7) | 57% | 51% | 3.3 s |
| Gemma 4 31B | Local (RTX 3090) | Yes (7/7) | 59% | 52% | 3.9 s |
| Claude Opus 4.8 (baseline) | Cloud API | Yes (7/7) | 55% | 42% | 2.7 s |
| GPT-OSS 20B | Local (RTX 3090) | No: missed 1 real fault | 84% | 77% | 1.6 s |
| Gemma 4 26B MoE | Local (RTX 3090) | Yes, but errored on 17 of 65 | 40% | 35% | 7.5 s |
The surprise was at the top of the table: the mid-sized local models didn’t just survive, they matched the cloud baseline at telling signal from noise and even edged past it. This routing decision, made without tools, has a low ceiling; there’s only so well anyone can guess from an alert’s text alone, and every competent model already clears that hurdle. The cloud model’s real advantage shows up later, in the open-ended investigation (SSHing into the server, running commands, chasing the cause), not in the snap routing judgment.
So I picked the smallest model that cleared the safety bar perfectly: a nine-billion-parameter model called qwen3.5:9b. It missed nothing real in the eval, exactly like the models three times its size, answers in about a second and a half, and needs only around 6GB of video memory, which leaves the card room for other LLM tenants.
The bake-off also found a bug I wasn’t looking for (the best kind!). My pipeline has a fallback for when the primary model is unreachable, a safety net that prevents alerts from silently vanishing. I had it pointed at a cloud model version-pinned months ago, and that model had since been retired from the vendor’s API, so the safety net had unknowingly been dead for weeks. The eval caught it only because the cloud baseline hit the same dead endpoint. Pin a model name and you’ve taken on a dependency someone can delete out from under you.
As excited as I am to run local models, my homelab doesn’t operate on enthusiasm (alone, anyway). The local router runs in shadow first: it sees every real alert and records the call it would have made, next to what my trusted pipeline actually did, and earns the decision only once the shadow log has agreed with reality for a while. Even this measuring broke once: the router logs UTC, the host had just switched to Pacific time, and the 7-hour skew made every prediction look like it never happened. Trust here is measured, not granted, and tests are ongoing.
The router is only the first tenant. The same card already runs the intelligence behind Immich, our family’s new photo home: the face grouping that sorts loved ones from strangers and the search that finds “dog at the beach” are both model inference that now happens at home instead of in a metered cloud call. A second, larger model is auditioning for a harder job, judging evidence rather than just routing it, on the same terms: measured, in shadow.
The cloud is still where the hard, open-ended reasoning lives. But the lab is full of small, bounded, constant decisions, and the bake-off convinced me those belong at home.