What AI detectors actually measure: perplexity and burstiness

Perplexity measures how surprising your word choices are to a language model. GPTZero, whose explainer popularised the term, puts it as how likely an AI model would choose the exact same words found in a document. If the next word is the one the model would have predicted, perplexity is low. Their published rule of thumb at the time was that a perplexity above 85 was more likely than not human.

Burstiness measures variation. Humans vary sentence construction - a long compound sentence, then a short flat one, then a fragment. Language models, in GPTZero’s phrasing, write with a very consistent level of AI-likeness, so their burstiness runs low.

A detector built on these computes them per sentence, usually by running your text through a smaller open model such as GPT-2 and asking how predictable it was.

Both statistics are properties of the text. Neither is a property of the person who wrote it. Nothing in the calculation has access to who typed anything.

Low perplexity means predictable word choice. Predictable word choice has many causes, and being generated by a model is only one of them.

• Writing in a second language. Staying inside vocabulary you are confident in is a sound strategy and produces exactly this signature - which is why seven detectors misclassified 61 percent of human-written TOEFL essays.
• Writing in a genre with fixed conventions. A methods section, a legal definition, a lab protocol. Formulaic phrasing is the correct register there, not a tell.
• Editing for clarity. Removing unusual constructions and irregular rhythm is what careful revision does. It lowers both statistics.

The third one has an uncomfortable implication: making your writing more polished can make it look more machine-made. That is not a hypothesis. On the HumanizerBench leaderboard, Grammarly posts the highest meaning-preservation score in the table - 94.3 - and a perfect 100.0 for readability, with a bypass rate of 0.0 - it edits conservatively and cleanly, and its output reads as machine to detectors precisely because of that.

This is the part most articles about detectors leave out. GPTZero states plainly at the top of its own explainer: as of autumn 2023 it no longer uses perplexity and burstiness, having migrated to a deep-learning architecture. The statistics survive as one component among seven signals rather than as the mechanism.

Turnitin has never published its method at all.

So the honest state of things: the standard explanation of how detectors work describes an approach the best-known detector abandoned, and the most widely deployed detector has never described its approach.

Which matters practically. If the mechanism were still two published statistics, you could reason about what would and would not trigger a flag. With undisclosed deep-learning classifiers you cannot, and neither can the person reading your score.

Not much about writing technique, and quite a lot about how much weight a number deserves.

A score is the output of an undisclosed classifier over statistical properties of text, trained on some distribution of writing that you were probably not consulted about. It is evidence that a document has certain textual properties. It is not evidence about a person.

If you want the consequences of that spelled out - what happens when different vendors run the same paragraph - see why the same text scores differently on every detector. If you have a report in front of you already, how to read a Turnitin AI writing report covers what that particular number is measuring.