Why the same text scores differently on every detector

The clearest published number comes from the Stanford study on non-native writers, which is usually cited for its headline false-positive rate. Look instead at the two figures underneath it.

• 89 of 91 TOEFL essays (97.80 percent) were flagged as AI by at least one of seven detectors.
• 18 of 91 (19.78 percent) were flagged by all seven.

That leaves 71 essays - four out of five in the corpus - where the detectors did not agree with each other. Same text, same seven tools, same run.

If you submitted one of those 71 essays, whether you were accused depended on which vendor your institution had bought.

Because they are not implementations of one measurement. They are different products.

• Different underlying models. A detector that scores perplexity against GPT-2 and one that runs a purpose-trained deep-learning classifier are asking different questions of the same sentence.
• Different architectures over time. GPTZero moved off perplexity and burstiness in autumn 2023; others did not. A comparison across vendors is partly a comparison across generations of method.
• Different training distributions. What counts as unremarkable human writing depends on whose writing the classifier saw. This is the mechanism behind the ESL bias, and it varies per vendor.
• Different thresholds and reporting rules. Turnitin now withholds any number below 20 percent; others report a figure at every level. The same underlying judgement is presented differently before you ever see it.

There is a further wrinkle that anyone comparing scores runs into. Several detectors do not really produce a continuous probability - they push output towards 0 or 1, so a passage is either fine or condemned with little in between.

HumanizerBench, which runs five commercial detectors over every sample, takes the median rather than the mean for exactly this reason: with binarising detectors in the set, one outlier moves the mean by as much as 0.25. Its methodology page says so outright.

So "the average of several detectors" is not a more reliable number than one detector. Averaging a scale with a near-binary is a statistically incoherent operation, and running your text through several tools to see what most of them say does not produce the reassurance it appears to.

Three things follow, none of them about writing technique.

• Only one report matters: the one your institution runs. Checking your draft against a different vendor tells you what that vendor thinks. Given the disagreement rate, it does not predict the score that will be attached to your name.
• A second opinion that clears you is not exoneration, and one that flags you is not proof. Both are one classifier’s output on a text whose properties several other classifiers read differently.
• Ask what the score is used for. Turnitin states it does not determine misconduct and provides data for an educator to judge. A number that seven tools would score seven ways is an input to a conversation, not a conclusion.

For what the number is measuring in the first place, see what AI detectors actually measure. If you already have a Turnitin report, importing it works from the passages it actually flagged rather than from the headline figure.