What AI detectors measure beyond perplexity and burstiness

Perplexity measures how surprising your word choices are to a language model. GPTZero, whose explainer popularised the term, puts it as how likely an AI model would choose the exact same words found in a document. If the next word is the one the model would have predicted, perplexity is low. Their published rule of thumb at the time was that a perplexity above 85 was more likely than not human.

Burstiness measures variation. Humans vary sentence construction - a long compound sentence, then a short flat one, then a fragment. Language models, in GPTZero’s phrasing, write with a very consistent level of AI-likeness, so their burstiness runs low.

A statistical detector can compute these values for sentences or passages by running the text through a reference language model and asking how predictable it was. The choice of model matters: a phrase that surprises one model may be routine to another because they were trained on different corpora and tokenise text differently.

Both statistics are properties of the text. Neither is a property of the person who wrote it. Nothing in the calculation has access to who typed anything.

Perplexity is not a percentage of AI authorship. It is usually a transformed average of next-token probabilities, and its absolute scale is model-dependent. Burstiness is even less standardised: vendors may use variation in sentence perplexity, length, syntax or a proprietary combination. Two tools displaying the same word can therefore be reporting different calculations.

A complete detector still needs a decision rule. It may aggregate sentence scores into a document score, require a minimum amount of prose, smooth neighbouring sentences and apply a threshold chosen to trade false positives against missed detections. Those product choices can change the reported result even when the underlying text features are identical.

Low perplexity means predictable word choice. Predictable word choice has many causes, and being generated by a model is only one of them.

• Writing in a second language. Staying inside vocabulary you are confident in is a sound strategy and produces exactly this signature - which is why seven detectors misclassified 61 percent of human-written TOEFL essays.
• Writing in a genre with fixed conventions. A methods section, a legal definition, a lab protocol. Formulaic phrasing is the correct register there, not a tell.
• Editing for clarity. Removing unusual constructions and irregular rhythm can make wording more predictable, although the effect depends on the detector and passage.
• Quoting or closely following a template. Ethics statements, reporting checklists and standard definitions deliberately reuse stable language. Original authorship does not make those phrases statistically novel.
• Writing a short sample. With less context, one formulaic paragraph can dominate the result. Some vendors therefore impose minimum-length rules rather than pretending every snippet is equally scorable.

This creates an uncomfortable possibility: conservative editing can improve readability without improving a detector result. One illustration is the HumanizerBench leaderboard, where Grammarly scored highly for meaning preservation and readability in the published test but had a 0.0 bypass rate. That benchmark does not prove why the detectors reacted as they did, and it should not be generalised beyond its samples. It does show that writing quality and detector evasion are separate evaluation axes.

Predictability is not plagiarism, misconduct or proof of automation. It can be a feature of clear, conventional and carefully edited human prose.

This is the part many detector explainers leave out. GPTZero states at the top of its own page that as of autumn 2023 it no longer uses perplexity and burstiness for detection after migrating to a deep-learning architecture. The same page later describes statistical methods as indicators in a broader system, which is a useful warning against reducing a changing product to two public terms.

Turnitin publishes the report's categories, eligibility rules and cautions, but not the model weights, training corpus or feature pipeline needed to reproduce its classifier. Other commercial vendors disclose different levels of detail. A product label such as "deep learning" identifies a family of methods, not a transparent measurement.

Perplexity and burstiness are a useful historical model of detector behaviour, not a reliable recipe for predicting a current commercial score.

Deep classifiers may learn combinations of syntax, semantics and long-range patterns that are hard to name individually. They may also be updated without old reports being recalculated. That makes the detector version, language model coverage and report date part of the evidence. A score without those details is less reproducible than it appears.

It also blocks a common inference: a passage does not receive a high score because it has one visible habit, such as repeated transitions or long dashes. Removing a single habit may change nothing, while ordinary edits elsewhere may move the result. Lists of supposed "AI words" are therefore poor explanations of a classifier and poor editing advice.

Model updates introduce distribution shift in both directions. A detector trained to separate 2023 human essays from 2023 model output may face later models, new paraphrasers and changing human writing practices. Recalibration can improve one genre while worsening another. Current performance therefore has to be measured on current outputs and held-out human writing, not inferred from a launch benchmark or the architecture's name.

Not much about writing technique, and quite a lot about how much weight a number deserves.

A score is the output of a classifier over properties of text, trained and calibrated on a particular distribution. Depending on the vendor, the displayed percentage may describe the share of eligible text classified into a category rather than a probability that the author used AI. It is evidence about how that system labelled that document, not direct evidence about a person.

• Check the denominator. Is the number sentence confidence, document confidence, or the proportion of qualifying prose? Those are different quantities.
• Check the operating conditions. Language, minimum length, supported file type and excluded material can determine what the model assessed.
• Check the version and date. A vendor update can change classifications without any change to the prose.
• Check the decision rule. Ask whether the institution uses the result for triage, a conversation, or a formal finding, and what corroborating evidence is required.

For writers, the durable advice is ordinary authorship practice: keep drafts, verify sources, preserve your own reasoning and edit for the intended reader. Trying to optimise a hidden classifier substitutes an unstable target for those goals.

For decision-makers, calibration matters as much as ranking. A classifier can rank generated passages above human ones reasonably well yet attach percentages that do not correspond to real-world probabilities. Before a score is used operationally, ask what the percentage denotes, how it was calibrated, on which population, and how performance changes at the chosen threshold. Without those answers, the decimal places add precision to the display, not certainty to the conclusion.

If you want the consequences of that spelled out - what happens when different vendors run the same paragraph - see why the same text scores differently on every detector. If you have a report in front of you already, how to read a Turnitin AI writing report covers what that particular number is measuring.