My agent broke down. The paperwork says it never happened.

I keep an experimental AI agent on a rented server. It wakes up on a schedule, does a small job with its own API key, and goes back to sleep. Last week I wrote about handing agents scoped credentials; this is that same experiment, a week later.

This week the key expired. The agent woke up, tried to badge in, got refused, and stopped. No alarm went off. I only found out because I was poking around on the box.

So I pulled up the audit trail — the record of everything the agent proposes and does, the thing that's supposed to let me stop poking around on boxes. It said: zero anomalies.

A broken agent, and a clean report.

The trail wasn't lying. It was blind. It records what the agent does — and the agent never got far enough to do anything. Refused at the door, no action taken, nothing to write down. The failure happened one step before the paper trail begins.

That's the sneaky part of good monitoring: the better your records, the more convincing their silence.

The fix took an afternoon. The agent now raises a flag the moment a run fails. And once a day, something independent calls the model with that same key, down the same path the real job uses — an actual live test, not a glance at yesterday's logs. If the key dies again, my phone buzzes that day, not two weeks later when the next scheduled inspection would finally have looked.

Here's my favorite part. My first version of that live test was broken in exactly the same way. I ran it from the wrong spot — outside the agent's proxy — and it reported a network timeout instead of the real expired-key error. A health check that couldn't see the one failure it was built to catch. I only noticed because the dead key was still around to test against. A check that has never seen the failure it checks for is a guess.

The house rule that came out of it, now written into my fleet's principles: anything a system leans on gets a live health check the day it's built — not after the first time it breaks quietly.

If you run anything on a schedule, ask it one question: what has to work before your first log line can even exist? That's where mine broke. I'll let you know what breaks next.