My reference model disagreed with itselfproject

Patchday reads every pending container update in my homelab once a week, flags the risky ones, and waits for my tap before anything installs. The flagging is one AI call per pull request: a risk level plus three yes-or-no flags for a security fix, a breaking change, and manual steps. Since the service shipped, those calls ran on a metered API key, on top of the subscription I already pay for every month. Patchday was built that way before I knew enough to have an opinion about it. Now I do, and I would much rather get more out of the subscription than buy API credits on top of it. It also looked like the easiest move in my fleet: same model, Claude Sonnet 5, same prompt, a different way in.

On September 25 the subscription path took over the weekly calls, with the API path left in the container as a rollback. A bake-off would decide whether the rollback could go. It classified twenty recently closed pull requests, and the pass bar had two parts: the subscription path returns a structured verdict every time, and the two paths agree on risk and all three flags for at least 18 of the 20.

The subscription path answered nineteen times. The twentieth call ran out of conversation turns before it produced a verdict. On the nineteen it did answer, it matched the API on risk every time and on all three flags fifteen times. Counted against the bar, that is fifteen full agreements out of twenty, so the new path failed both parts of the test as written.

The API path had also answered every question twice, because an earlier attempt failed on my side, so I compared the old path with itself. It agreed on eighteen of the twenty, flipping once on a security fix and once on a breaking change. I thought I had restricted the task far enough (a narrow, well-defined question, a forced answer format, deliberate guardrails) that the model’s nondeterminism wouldn’t reach it. It reached it anyway.

Eighteen of twenty agreements cleared the bar, barely, and that is the problem with it. When the reference only just meets the bar against itself, a replacement exactly as good as the old path lands on either side of it by chance.

So I read the four disagreements one at a time. Two were on the pull requests where the API had flipped, one on the security-fix flag and one on the breaking-change flag, and each time the subscription path matched one of the API’s two answers. The other two were extra flags: manual steps on one update, and a breaking change plus manual steps on another. In that run, the new path never dropped a flag the API had called both times.

For Patchday, that direction matters more than the count. A verdict there is advice, and I still tap Install myself. An extra flag costs me some reading and maybe a week’s wait. A missing flag invites me to install something that needed my attention first. On October 1 I approved a bar built on that: a verdict on every call, and no dropped flag on any pull request where the API agreed with itself.

The rerun came after my fleet moved to Claude Sonnet 5.5 on October 10: both paths, twice over the same twenty pull requests. Every call returned a verdict. Full agreement fell to nine of twenty and fourteen of twenty, and each path matched itself on only sixteen, so the new model was noisier. Noise didn’t explain the misses. On a Gotenberg update and a changedetection.io update, the API flagged a breaking change in both runs and the subscription path flagged it in neither. Over both runs, the API called a breaking change the subscription path didn’t nine times, on six pull requests, and never the reverse. That was the miss the new bar existed to catch.

On October 16 I chose to retire the API path despite the failed bake-off. I wanted to stop paying separately for those calls, and I accepted the subscription path’s misses while I investigated them. The API path came out of the container the next day, and I asked Claude Code to find out why the paths disagreed. My guess was effort, the setting for how much the model thinks before it answers. Claude Code pointed the subscription path at a local stand-in server that recorded its request instead of answering it: same model, and the same high effort the API code had pinned. What differed was the wrapping. The subscription path runs through Anthropic’s agent SDK, which adds an identity line and a note about its environment, and collects the verdict through a tool call. The API path sent the prompt bare and had the API enforce the answer’s shape.

The verdicts show the split. Patchday’s answer format defines a breaking change as a config rename, a dropped environment variable, a database migration, or a changed default. In the first run, the subscription path’s notes on the Gotenberg update listed two new defaults, and its summary said nothing should need changes in a typical Paperless setup. It set the flag by whether the update would break a setup like mine. The API path applied the definition as written, and by that definition it was right. What tipped the API path stays open: on Sonnet 5 both paths had left those two updates unflagged, and the API path then changed models and answer formats in one move, which can’t be untangled now that it’s gone.

The same investigation tried a sharper wording on the subscription path, one call per update: breaking whenever the notes name a changed default, even when my homelab probably doesn’t use the setting, with the risk level carrying whether it matters here. Both updates came back breaking, at moderate risk. On October 17 I approved that wording, with tightened security added to the list, and it is the definition Patchday runs on now.

Patchday now runs subscription-only with the sharper definition. In the two targeted checks before it shipped, one call per update, that wording flagged both updates as breaking changes at moderate risk. That is narrower evidence than a passing bake-off, and I still review each verdict before tapping Install.