Convincingly wrong

While publishing a new version of a website, something seemed stuck: the test environment appeared to serve old code. I asked the AI that handles our deployments. It produced the kind of diagnosis you'd hope for — named a mechanism in the hosting platform, explained coherently how a "stale state can get pinned", and derived a fix: stop and restart the environment so it picks up the fresh version.

It sounded assured. It was wrong — on both levels.

Two errors, one outage

First, the diagnosed mechanism didn't exist in our setup at all. The observation "test environment shows old code" had a far more mundane cause: an earlier step had already pushed the new version to the live site — after which the test environment was, entirely correctly, showing the previous one. There was nothing to repair.

Second, the derived "fix" wasn't just unnecessary — it was dangerous. Through an unlucky chain of circumstances, the stop command hit not the test environment but production. The real, public website was gone for 27 seconds. No catastrophe — but an outage caused by repairing a problem that never existed.

Shortly afterwards, the mirror image of the same pattern: on the next deployment, the system reported "swap failed" and the command exited with an error code. This time the message was the false part — the new version was already live and healthy. Anyone who had trusted the error and "fixed things up" would once again have been operating on an intact system.

Why this is the dangerous property

An AI diagnosis sounds equally assured whether it's right or wrong. There is no hesitation in the voice, no "I'm not sure about this part" that reliably tracks the actual uncertainty. Language models generate plausible explanations — that is their core competence. And plausibility feels exactly like truth, from the inside and the outside. The difference only shows up on contact with reality.

Add a human amplifier: a detailed, terminologically confident explanation reads as more verified than a vague one. But detail is no quality seal in a language model's output — it can describe a perfectly real mechanism that simply doesn't apply to your situation.

The discipline against it

Those incidents hardened into three rules we've worked by since:

1. Measure against reality, not against the explanation. Our websites expose a public address that reports which version is actually running. Any claim about the live state — from the AI, from a tool, from an error message — gets checked there before anyone acts. Measure first, believe second.
2. Separate diagnosis from intervention. Listening to an explanation costs nothing. An intervention that changes system state — restarting, deleting, reconfiguring — needs its own evidence that this specific intervention is warranted. "The diagnosis sounds convincing" is not evidence.
3. Status messages are claims. The reverse direction holds too: "failed" doesn't necessarily mean failed. Success and failure are judged by the outcome, not by the return code.

The remarkable part: under these rules the AI doesn't work worse — it works better. It now carries the rules in its own notes (yes, it keeps notes) — including a warning about its own misdiagnosis back then.

What to take away

The pattern isn't limited to servers. When an AI explains why your printer won't print, what a rash means, or why your contract can be terminated: the eloquence of the answer tells you nothing about its correctness. Treat AI diagnoses as well-argued hypotheses — a starting point for verification, not a substitute for it. And for anything hard to reverse, the workshop rule applies: measure first, act second.