My red pen has a new studentproject

Somewhere in the past decade my inbox stopped being mail and became a feed: receipts, alerts, promotions, and buried between them, the occasional human who actually needs me. This was mostly self-inflicted. Sorting that feed is a bounded, constant judgment call, exactly the kind of job I’ve been moving onto the homelab’s GPU (an earlier post covers why). So every 15 minutes, Mailroom reads whatever arrived and files it under one of a few labels: bulk, FYI, actionable. A 14-billion-parameter local model makes the call, so no email leaves the house to get read, and the agent’s powers are earned, never granted: it opened its career labeling and doing nothing else. Labels were never the end goal, though: I want an inbox that eventually runs itself, and an agent gets there by building a record I can check. This post is that record: how the agent gets better, how I prove it got better, and what the proof bought it.

A mid-size 14B local model is not a frontier model, and I didn’t pretend otherwise. Mailroom opened as an understudy: a frontier model labeled the same mail in parallel, and the local model’s verdicts only had to match. At first they matched 72.9% of the time. Closing the gap taught me how this particular model actually learns: worked examples, not prose. I wrote careful rules (“delivery notices are FYI”) and the model sailed right past them; turning the same rules into worked examples, each a real subject line with a verdict and one line of reasoning, moved agreement to 85% on the same 300-email exam.

The remaining misses are where the interesting mistakes live: a doctor’s message filed as FYI, a SIM-swap security notice called bulk. My entire interface for fixing them is Gmail itself. When I catch a wrong label, I set the right one and add a “mislabeled” marker; no retraining ceremony, no feedback form. The marker is my red pen, the same one that’s been training this blog’s ghostwriter all along.

Every Sunday at noon (timed for when the solar panels are carrying the house and a multi-hour GPU run will go unnoticed), Mailroom studies the week’s red ink. It hands my corrections to a frontier model (currently Claude Opus 4.8) to rewrite the classifier’s instructions; that is the system’s only cloud call, and it sees only mail I explicitly flagged. The rewritten prompt then has to survive two checks. First, an injection guard: if any 40-character run from an email body appears verbatim in the candidate, it’s rejected, because a stranger’s email must not be able to write itself into my system prompt. Second, a regression gate: the candidate must beat the current prompt on my corrections and hold steady on a frozen 300-email exam. Pass both, and it adopts itself: version bump, git commit, a report to Discord. Last Sunday was the first run at real scale. 29 corrections went in; the new prompt scored 69% on them where the old one managed 10%, and improved on the regression set too.

Those same 29 corrections tripped a second threshold I’d set weeks earlier: enough data to justify fine-tuning. By evening, the GPU that classifies my mail had trained its own replacement: a LoRA adapter (a thin trainable overlay on a frozen model, cheap enough for a home GPU) trained on 529 recent messages the current prompt labeled for itself, plus my corrections, upweighted. About 2.5 hours on the RTX 3090, and the email bodies were deleted the moment training finished. The adapter serves under its own model name while the untouched base keeps answering for the voice assistant and the log alert router; that separation is the firewall against fine-tuning for one job degrading every other (“catastrophic forgetting,” the aptest term in machine learning I’ve come across).

What broke, repeatedly, was the measurement, and never the model. Three different graders took turns judging the new adapter, and each was broken in its own way. The training container’s built-in check came first: it passed the adapter, but a bug had left it with only 3 test cases, so it was a rubber stamp, approving everything. I replaced it with a referee built that same morning: replay the 60 emails the old model had gotten wrong and see whether the new one fixes them. It did, convincingly, and the swap shipped on that result. But that referee only asks one question, “did the new model fix the old mistakes?”, and it structurally can’t see the opposite failure: new mistakes on emails the old model was getting right.

So I built the exam that asks both questions: replay all 317 recent emails and compare the two models on everything. At first that run looked worse (more dangerous misses, not fewer), until I checked the flagged misses by hand and found half of the ones that mattered were errors in my answer key, not in the model (it had filed a university alumni mailing as junk; my golden label was the wrong one). With the answer key corrected: 2 dangerous misses across 317 emails. The adapter had been good all along; it took three graders to prove it. Every grader turned out to need a grader, including me.

So the loop is closed, and it runs weekly whether I tend it or not: my corrections become a better prompt, and when they pile high enough, better weights. And this week, that hard-won measurement paid out. With its archive verdicts measured at 95% precision and a review of over a thousand of its calls backing that number up, Mailroom earned its first real privilege: bulk mail now files itself out of my inbox without me. Every other label stays advisory, and the next privilege gets earned the same way.