Workshop notes

The AI that looked like malware

My antivirus reported a trojan: "This program is dangerous. It executes commands from an attacker." The attacker was my AI — and all it wanted was to start a test server. Why harmless AI code can set off alarms.

One June evening, an AI was verifying website changes for me. To do that, it needed to briefly start a local test server — routine work. To keep a console window from flashing across my screen, it picked what seemed to it the tidy option: launch the server as a process with a hidden window, via script.

Seconds later, Microsoft Defender struck. Classification: trojan, severity "severe". Justification, in the system's words: this program is dangerous, it executes commands from an attacker. The process was removed and the AI's task failed — from its point of view, without a trace: command sent, nothing came back. And my screen showed a malware warning, triggered by my own tool.

[BILD: screenshot of the Defender alert (threat removed, severity severe), if reconstructable. Target path /images/articles/defender-falle/.]

Not a false alarm, strictly speaking

What makes this incident worth telling: the antivirus wasn't being paranoid. It did exactly what it's for.

Modern protection software doesn't just match known signatures; it judges behaviour. And this behaviour — a scripting engine, started with relaxed policies, spawning an invisible child process — happens to be the textbook pattern of malware quietly loading its tooling. This is precisely what it would look like if a real attacker were gaining a foothold on my machine. The internal detection name spelled it out: roughly, "PowerShell, hidden, sub-process execution".

The AI meant no harm. But intent isn't stored in process memory. A heuristic evaluates what happens, not what for — and by everything observable, this was indistinguishable from an attack. The message "executes commands from an attacker" wasn't even entirely wrong: the process really was executing a third party's commands. It's just that the third party was working for me.

Why AIs wander into these patterns

An AI picks its means by expedience: window in the way? Hide the window. It knows plenty about malware techniques in the abstract — but in the thick of a concrete task it optimises for "works", not for "looks trustworthy to a watchdog". Experienced system people run that second check instinctively; they avoid certain tricks not because the tricks fail, but because they look suspicious.

The fix was suitably unspectacular: the hidden-window route was banned. Since then the AI uses the visible, purpose-built background-process feature of its own tooling — same effect, no alarm, and it ran flawlessly in the very same session. The incident now lives as a rule in its notes: no hidden processes, no clever circumventions — take the official paths. It's the same stance as our agent guardrails: a workaround that looks like an attack is not a good workaround.

What to take away

First, the reassuring part: if your antivirus flags an AI-generated script or program, it does not automatically mean the AI slipped you something malicious. Behavioural detection flags patterns — and AI code occasionally lands in suspicious patterns because it is built for expedience rather than convention.

Second, the sobering part: you must not rely on that. The same alert could be real next time. AI-generated code deserves review before it runs — precisely because intent is invisible from the outside. The antivirus stays on, its warnings get read rather than dismissed, and when in doubt: understand what ran, then approve. The watchdog that mistook my AI for a burglar was doing its job. Those are exactly the watchdogs I want to keep.

/compact — the essentials, if context is running low:

An AI launched a local test server as a process with a hidden window — Microsoft Defender classified it as a trojan and removed the process. Not a false alarm, strictly speaking: behavioural detection judges what happens, not what for, and an invisible child process is the textbook pattern of malware — AI code occasionally lands in suspicious patterns because it is built for expedience rather than convention. The takeaway: no hidden processes and no clever circumventions, but the official, visible paths — and because the same alert could be real next time, AI-generated code deserves review before it runs.

An unhandled error has occurred. Reload 🗙

Rejoining the server…

Rejoin failed… trying again in seconds.

Failed to rejoin.
Please retry or reload the page.

The session has been paused by the server.

Failed to resume the session.
Please retry or reload the page.