Do AI detectors actually work?

There’s a particular kind of email that teachers have learned to dread. A student has been accused of using AI, the evidence is a percentage from a detection tool, and the student says they wrote every word. Who’s right? Very often, nobody can tell. That’s the problem.

How detectors guess

Most AI detectors don’t find anything. They estimate. The usual method measures how predictable a piece of writing is: how often each word is the one a language model would have picked anyway. AI text tends to be very predictable. So, unfortunately, does plenty of human writing. A clear report is predictable. So is a careful essay by someone writing in their second language. The detector isn’t spotting a machine. It’s spotting plainness, and then making an accusation about it.^*

What the evidence says

OpenAI built its own detector in early 2023 and withdrew it that July. By the company’s own figures it caught 26% of AI-written text and wrongly flagged 9% of human writing. If the makers of ChatGPT couldn’t reliably spot ChatGPT, it’s fair to ask what a free website is doing differently.

The same year, Stanford researchers ran essays written by non-native English speakers for the TOEFL exam through seven popular detectors. On average, 61% were labelled AI-generated. Every one had been written by a person.

The tools have improved since, and a 2026 study of Czech essays found little sign of the same bias. The people who have to act on the results are still backing away, though. Yale won’t let detector scores be cited in formal misconduct cases. Johns Hopkins treats a flag as the start of a conversation rather than as evidence. The University of Waterloo switched off Turnitin’s AI detection after internal tests flagged fully human work as 100% AI.

The editing problem

Even a good detector struggles with the way people actually use AI. Very little writing is purely human or purely machine any more. Someone drafts with ChatGPT and rewrites half of it, or writes it all themselves and asks Claude to tidy the grammar. What percentage of that is “AI”? The question has no sensible answer, so the tools give a confident one instead.^†

What about watermarks?

Watermarks are a different animal. Rather than guessing from style, an AI company hides a statistical pattern in its own model’s word choices and checks for it later with a secret key. That’s far more reliable for one model and useless for any other. Since August the EU has required providers to mark their text this way, but the pattern fades as words get changed, and the checkers are mostly reserved for researchers and authorities. More on how text watermarking works.

So do they work?

As a hint, sometimes. As proof, no. If you’re judging someone’s work, judge the work and ask them about it. A person who wrote something can usually talk about it for five minutes. A percentage can’t.

1. * Roughly like identifying burglars by checking who owns a torch. ↩
2. † Confidence is cheap to manufacture. It is, if we’re honest, the thing computers were always best at. ↩