Letting the on-call engineer actproject

A failed SSH login carries whatever username the visitor typed. A web request carries whatever path a stranger aimed at it. My alert investigator’s raw material is logs, and logs are partly written by strangers, which means anyone on the internet can put words in front of my agent, and words in front of an agent are an attack surface: craft the right text in a log line and you’re not attacking my server, you’re lobbying my AI (aka prompt injection). Wire an agent that reads hostile text to something that can change my systems and you’ve run a cable from the internet straight to my configuration. Every guardrail in this post exists to cut that wire.

The agent in question is the investigator from a few posts back: when something trips a rule, it SSHes in read-only, runs its commands, and posts a verdict to Discord. That post ended with a contract I was still drafting: it will propose, I will approve, and nothing will execute without a human’s fingerprint on it. The contract is signed now, because diagnosis alone kept leaving me homework. The investigator would correctly conclude “this alert is noise, here’s the config change that would silence it,” and then the change would sit in my backlog behind everything else. The obvious next step was letting it act. “Never half-ass it” from an earlier post makes a reappearance!

So the design starts with a hard rule: the investigator agent proposes, but never executes. When it wants to make a change, it files a proposal into a database and stops. The execution lives in a different program entirely, one with no AI in it, which only performs a short list of pre-approved operation types against the API for Graylog, my alerting system. The agent cannot reach the API; the executor cannot be talked into anything, because it doesn’t listen to language at all.

Enter me, the human in the loop. The approval screen is itself a security decision. The Discord message renders the literal change: this rule, this field, this exact value. The agent’s persuasive English explanation is displayed too, but as commentary, because the explanation is the one part an attacker could shape. I approve the mutation, not the argument for it. If the reasoning is brilliant and the change is wrong, the change is what matters; if the reasoning was injected nonsense but the change is harmless, the render shows me that too.

Approval itself is gated by identity. Only allow-listed accounts can approve (my family Discord server has other people in it), and the claim is atomic, so a proposal can only execute once no matter how many times it gets clicked. Even the click mechanism earned its place through failure: I built approval buttons twice, and both times Discord’s button registration failed silently after a restart. Come to find out, emoji reactions survive restarts and carry the reactor’s identity, so I now LGTM changes with a 👍. Kind of geeky, but it works.

The last guardrail is scope. The agent can propose exactly one category of change: alerting configuration, essentially the “rules” about what counts as noise. It cannot propose commands, container changes, or anything with SSH in it. If every guardrail above failed at once, the blast radius is that my alerts get quieter, which results in an inconvenience, not a breach.

A week in, the rhythm is already what I hoped for: the investigator closes an alert as noise, proposes the rule that silences that noise forever, and I approve it from my phone with a tap. The pattern (propose, render literally, gate by identity, execute out-of-band) is the reusable part, and it’s about to get a bigger test: the next agent I want on staff reads my email, which is a firehose of exactly the kind of text nobody should trust. Same contract, higher stakes.