Why the same text scores differently on every detector

The clearest published number comes from the Stanford study on non-native writers, which is usually cited for its headline false-positive rate. Look instead at the two figures underneath it.

• 89 of 91 TOEFL essays (97.80 percent) were flagged as AI by at least one of seven detectors.
• 18 of 91 (19.78 percent) were flagged by all seven.

That leaves 71 essays - four out of five in the corpus - for which the seven binary decisions were not unanimous. The published counts demonstrate disagreement about classification, although they do not tell us how far apart every underlying score was.

For those essays, the tool selected by an institution could change whether a flag appeared. That is a product-dependence problem, not evidence that one particular detector was always wrong.

The date matters. The study captured seven products in early 2023, and several have since changed models, thresholds or reporting interfaces. It should not be used as a current league table. Its durable result is methodological: detector outputs are not interchangeable, so a score from one vendor cannot validate or cancel a score from another without a new, controlled comparison.

Even a repeat run from the same vendor may differ if the service updated silently, the account has different features enabled, or the document was parsed differently. Reproducibility therefore requires more than saving the prose: retain the original file, report, date, product tier, language setting and any available model version. A screenshot of one percentage without that context is difficult to audit later.

Because they are not implementations of one measurement. They are different products.

• Different underlying models. A detector that scores perplexity against GPT-2 and one that runs a purpose-trained deep-learning classifier are asking different questions of the same sentence.
• Different architectures over time. GPTZero moved off perplexity and burstiness in autumn 2023; others did not. A comparison across vendors is partly a comparison across generations of method.
• Different training distributions. What counts as unremarkable human writing depends on whose writing the classifier saw. This is the mechanism behind the ESL bias, and it varies per vendor.
• Different thresholds and reporting rules. Turnitin now withholds numerical scores from 1 to 19 percent; others report a figure at every level. The same underlying signal can be presented differently before you ever see it.
• Different units. One percentage may mean document confidence, another the share of sentences labelled AI, and another the share of qualifying prose. Equal-looking numbers need not answer the same question.
• Different preprocessing. Tools may strip references, ignore bullet points, split sentences differently or truncate long documents. They can therefore score different text while appearing to receive the same file.

Length compounds these differences. A detector that needs several hundred words can use document-level patterns; a browser tool accepting one paragraph may rely on local features. Language support also matters: a model calibrated for English cannot be assumed to behave the same way on translated passages or a bilingual reference list.

The vendors' own definitions show why equal-looking percentages cannot be compared until you know the unit. Checked on 28 August 2026:

Turnitin therefore reports a share of an eligible subset of the document, while the other three examples report confidence around a class label. The same symbol, percent, is doing different mathematical jobs. Converting one vendor's number into another vendor's scale is not a second opinion; it changes the question.

GPTZero's own public pages make the unit problem visible without comparing two companies. Its API documentation and its web-detector page describe two different outputs, both with percent signs. Checked on 28 August 2026:

These can be separate output fields rather than a contradiction. The practical rule is to name the product surface and field. A bare phrase such as "GPTZero returned 30%" does not identify which quantity the number represents.

The same unit mismatch appears in two current Chinese first-party descriptions. The quoted wording below was checked on 28 August 2026.

Those statements attach the same visual symbol to different quantities: probability of presence versus suspected-text share. A value of 30% from one page cannot be placed beside 30% from the other without first changing the unit, and neither page publishes a conversion rule.

There is a further wrinkle when comparing scores. Some detector outputs in a benchmark cluster near 0 or 1, while others occupy more of the scale. That does not necessarily mean the underlying model has no continuous score; thresholding, rounding and the user interface can make a continuous signal look binary.

HumanizerBench, which runs five commercial detectors over every sample, takes the median rather than the mean for exactly this reason: with binarising detectors in the set, one outlier moves the mean by as much as 0.25. Its methodology page says so outright.

A mean across such outputs is mathematically possible, but its interpretation is unclear when the inputs use different units and calibration. A median reduces the influence of one extreme value; it does not reveal ground truth or make the component detectors independent. HumanizerBench is a published product benchmark, not an institutional validation study, so its aggregation choice should be read in that scope.

Majority voting has a similar limitation. Five correlated tools trained on overlapping corpora are not five independent witnesses. Agreement may reflect shared blind spots, while disagreement may reflect nothing more than different cutoffs. Without labelled human and generated test sets that match the target population, there is no principled way to turn the votes into an authorship probability.

Three things follow, none of them about writing technique.

• For a live institutional process, identify the report actually being used. Checking a different vendor tells you what that vendor thinks; it cannot reproduce the institution's tool, settings or version.
• A second opinion that clears you is not exoneration, and one that flags you is not proof. Both are one classifier’s output on a text whose properties several other classifiers read differently.
• Ask what the score is used for. Turnitin states its model may misidentify text and should not be the sole basis for adverse action. A detector result is an input to review, not a misconduct finding.
• If you are comparing tools for research, hold the conditions constant. Use the same version, full input, language, date and labelled corpus; record failures and excluded text as well as the displayed number.

For an appeal, competing screenshots usually add noise. Process evidence - drafts, notes, source annotations, revision history and the ability to explain decisions - addresses authorship more directly than another opaque classifier. For an educator, the disagreement is a reason to design a corroboration process before a difficult case arrives, not after.

For procurement, compare error by subgroup and genre rather than asking which vendor reports the highest overall accuracy. Require a documented update policy, an audit trail for report versions, clear eligible-text rules and an export that students can meaningfully answer. Disagreement becomes less dangerous when the institution knows exactly what the tool contributes and has defined evidence that can override it.

For what the number is measuring in the first place, see what AI detectors actually measure.