What a 1% false positive rate means when a university submits 75,000 papers

A false positive rate is a property of a population, not of your paper. Vanderbilt University put numbers on that in August 2023: it had submitted 75,000 papers to Turnitin in 2022, so if the AI detector had been running then, a 1% false positive rate would have meant "around 750 student papers could have been incorrectly labeled". It disabled the detector. Two conditions ride along with that sentence and both get dropped in the retelling: the detector was not running in 2022, which is why the source says "if", and the vendor's own 1% figure carries a qualifier that its own next sentence throws away.

Vanderbilt's Office of Learning and Innovation published a post on 16 August 2023 titled "Guidance on AI Detection and Why We're Disabling Turnitin's AI Detector". The decision itself is stated plainly: "Vanderbilt has decided to disable Turnitin's AI detection tool for the foreseeable future."

The passage worth carrying around is the arithmetic:

"At the time of launch, Turnitin claimed that its detection tool had a 1% false positive rate (Chechitelli, 2023). To put that into context, Vanderbilt submitted 75,000 papers to Turnitin in 2022. If this AI detection tool was available then, around 750 student papers could have been incorrectly labeled as having some of it written by AI."

Nothing clever is happening there. It is one per cent of seventy-five thousand. But it converts a percentage into a number of documents, and that conversion is the whole reason error rates are hard to think about.

Now read the second sentence again, starting at the word if. The detector was not running when those 75,000 papers were submitted, which is what "if this AI detection tool was available then" means. So 750 is not a count of anything that happened at Vanderbilt or anywhere else. It is what one per cent of one university's real annual volume looks like written out. That is a smaller claim than the one the figure usually gets recruited for, and a much easier one to defend.

For a vendor, one in a hundred is a specification. For an institution, it is a yearly caseload. For you, it is neither — you are one document, and the rate says nothing about whether yours is the one.

That last point cuts both ways, and it is where people overreach in both directions:

• It does not mean your flag is probably wrong. A 1% error rate on human-written papers is entirely compatible with most flagged papers being correctly flagged, depending on how many were AI-assisted to begin with.
• It does mean "the tool is 99% accurate, so you must have done it" is not an argument. A rate describes a population; it cannot tell you which member of that population is in front of you. And one per cent of a 75,000-paper year is hundreds of documents, so being in that band is unremarkable rather than exotic. Hundreds of documents is not hundreds of accusations, though, and the distance between those two is where most bad versions of this argument live.

The useful posture is neither "the detector is broken" nor "the detector is right". It is that a population statistic cannot adjudicate an individual case, and the vendor agrees in writing: its FAQ tells instructors not to treat the AI writing indicator's percentage as "the sole basis for action or a definitive grading measure".

Here is a detail that matters if you are ever going to quote the 1% figure at anybody.

Turnitin's FAQ states the claim like this: it strives to keep its false positive rate — "incorrectly identifying fully human-written text as AI-generated" — "under 1% for documents with over 20% of AI writing."

For documents with over 20% of AI writing. That is a substantial limit on the claim, and it does not survive the vendor's own restatement a sentence later, which reads: "In other words, we might flag a human-written document as AI-written for one out of every 100 fully-human written documents."

The two sentences are not equivalent, and the second is the one that travels. If you use the figure, use the qualified version — partly because it is accurate, and mostly because someone on the other side of the table may know the qualifier is there.

The post is from August 2023. Part of it is arithmetic, which does not age. Part of it is a claim about what the vendor had published, which does. Sorting those before you quote anything is most of the work.

The arithmetic is safe to repeat, with the if attached. One per cent of a large submission volume is a large number of documents, and nothing since 2023 touches that.

The concern about non-native English writers is safe to repeat, and it does not rest on this post at all. There is research behind it, and we went through what that research actually measured in AI detector bias against non-native English writing.

The transparency complaint is the one to check before you repeat it. Vanderbilt wrote that "Turnitin gives no detailed information as to how it determines if a piece of writing is AI-generated or not". Today's FAQ does describe how a score is assembled: sentences are extracted and segmented into overlapping sections, each segment is classified and given a value between 0 and 1, qualifying sentences inherit their segment's score, overlapping scores are pooled, and the pooled sentence scores are aggregated into a document score. Assert in a meeting that nothing has ever been published about how it works, and someone can put that paragraph in front of you.

Read the sentence that follows it in the post, though, before deciding the objection was answered, because the objection is narrower than "tell us how it works". The complaint is that the tool "looks for patterns common in AI writing, but they do not explain or define what those patterns are". A description of how segment scores are combined is not a description of the patterns. The published mechanism and the published objection are about two different questions, and both can be quoted accurately on the same day.

What we cannot tell you is when that FAQ passage first appeared. We have not dated it, so nothing above should be read as the vendor publishing it in answer to this post.

One more qualifier, and this one belongs to Vanderbilt. The post describes the feature as arriving with "less than 24-hour advance notice, no option at the time to disable the feature". At the time is in the original sentence. Quote it without those three words and you have converted a statement about the launch into a standing claim about the product. That is the same move as dropping "over 20%" from the 1% figure, just made by the other side of the argument.

1. Cite the date. "In 2023, Vanderbilt disabled the tool and published this reasoning" is checkable and defensible. "Universities have abandoned AI detection" is neither.
2. Say what the arithmetic is an illustration of. It travels because anyone can redo it with their own institution's volume, and it makes a real point: a small percentage still produces a caseload somebody has to review fairly. What it does not do is estimate accusations. Not every submission is eligible for a score, nobody outside the institution knows how much AI use there actually was, and a flag does not have to become a charge. We take that apart at more length in is 20% AI too high.
3. Do not claim it proves anything about your document. It establishes that being wrongly flagged is an ordinary event at scale, which is a different and more modest claim.
4. Check whether your own institution has published anything. Look in the academic integrity policy and in whatever the teaching and learning centre puts out, not only the news page. A local statement outranks a distant one in every conversation you are likely to have.
5. Keep it out of your opening. Process and provenance persuade first; statistics are what you reach for when someone argues that the number is self-evidently conclusive.

If it does, the report in front of you is the object that matters rather than any of the above. What the percentage is a statistic of, and why the highlights and the number can disagree, is separate ground, covered in reading the AI writing report line by line.

HumanPen enters only at the point where specific passages are marked and some of them need to change: the report sets the boundary, and unmarked paragraphs keep the wording you already have.

None of that is an answer to a false positive. If your position is that you wrote it, rewriting it is the wrong move, and what has actually persuaded reviewers is a different subject entirely: the evidence that has moved a panel.

Does a 1% false positive rate mean my flag is probably wrong? No. It is a rate across a population of human-written documents and says nothing about an individual case. It does mean that at institutional volume, wrongly flagged papers are an ordinary occurrence rather than a freak event.

Where does the 750 figure come from? From Vanderbilt's own post of 16 August 2023, applying the 1% figure to the 75,000 papers it submitted to Turnitin in 2022. Note the conditional: the detector was not running in 2022, so the sentence reads "if this AI detection tool was available then". It shows what one per cent of that university's volume looks like written out. It is not a record of 750 papers being wrongly flagged, and it is not a forecast for anyone else's institution.

Is the 1% claim unconditional? No. Turnitin's stated target is under 1% "for documents with over 20% of AI writing". Its own restatement in the following sentence drops that qualifier, which is why the unqualified version circulates.

Have universities generally turned AI detection off? That is not something this source supports. It documents one institution's decision in 2023, with its reasoning. Whether your institution uses it is a local question with a local answer.