17. The AI-Moderation Wave

The new generation of tools brought AI to the defense.

If the top of the distribution walks straight through a rule, the obvious next move is to stop writing rules. For two decades the moderation stack had been built out of them. A pattern matched or it did not; a link appeared on a blocklist or it did not; an account tripped a rate limit or it did not. The previous chapter walked the canon of that approach and credited it honestly: rule-based defense still catches the bottom of the threat distribution, the lazy floods and the recycled scripts, cheaply and reliably. But the same chapter ended at the ceiling, and the ceiling is unavoidable. A rule is a specification. The attacker reads the specification — sometimes by inspecting the open-source bot, more often by simply observing which messages get deleted — and builds the one input that satisfies it. Rules are brittle precisely because they are legible. You can route around anything you can read.

So when generative language models became cheap enough to call on every message in a busy chat, a wave of defenders did the obvious thing. If the attackers were using large language models to write their lures — and by 2025 they unmistakably were — then the defense should use the same class of model to read them. Where a regex sees a string, an LLM sees meaning. It can be handed a message and a paragraph of context and asked, in plain English, "is this a scam?" — and it will answer with something that looks a great deal like judgment. This was a real advance, and it is important not to be cynical about it, because it genuinely closed gaps that rules could never close. It is also, as this chapter will argue, the last move available to a strategy that was doomed from the start, because it is the most sophisticated possible way of inspecting an artifact, and the artifact is now indistinguishable from the thing it imitates.