How to check an AI humanizer's "preserves your formatting" claim

A formatting claim can only be checked if it names four things: which element, in which file layer, out of how many, counted how. "Preserves your citations" names none of them. "Markdown headings present in the input survived in 291 of 552 cases" names all four, and you can disagree with it.

The rest of this is the test, applied to the only open dataset I could find in this category — which turns out to be published by a competitor, and is still the most checkable thing here — and then to our own numbers, gaps included.

The layer question is the one that gets skipped, and it is the one that decides whether a number means anything for a thesis. A hash at the start of a line and a Heading 2 style in `word/document.xml` are different objects with different failure modes. A tool can be perfect at one and hopeless at the other.

Start with who publishes it, because a claim inherits the interests of whoever is making it.

The dataset everyone in this category quotes is HumanizerBench, and its about page carries a sentence most benchmarks would bury: "We're operated by WriteHuman, which sells one of the humanizers on this leaderboard." Every page on the site ends with the same admission in the footer, and WriteHuman holds first place in the cycle quoted below. So it is not an independent benchmark, and a citation that calls it one has skipped a step.

What it is instead is the rarer thing. It publishes the files. Every input, every output and every detector verdict from the August 2026 cycle is in a public repository under CC BY 4.0. A benchmark run by a competitor that hands over the raw data can be checked by a stranger; a neutral-sounding round-up with nothing to download cannot. Both halves of that have to travel together, and the second half is why the numbers below are worth anything.

The cycle is 12 tools by 33 samples, 396 completed tests. Counting elements that were present in an input and came back in the matching output:

Now the four questions. Element: named. Denominator: published. Counting method: a rule you can apply yourself, since a Markdown heading is a line beginning with a hash and a bold span is text between double asterisks, and survival is capped at what the input actually had, so no tool earns credit for inventing a heading that was never there. Download the cycle, apply that rule, and you will either land on these two numbers or find out where I went wrong. Layer: Markdown characters in plain text. Everything in that dataset was pasted into a box and pasted back out; no `.docx` was involved at any point.

That last line is not a criticism of the benchmark, which never claimed otherwise. It is the thing you have to carry with you when a vendor quotes it at you.

Same four questions, pointed at us. And the first thing to say is the thing that decides how much weight this deserves: we measured our own output, on files we chose, with our own script. It is a vendor measuring itself.

We took three before-and-after pairs, opened each file as a zip and walked `word/document.xml`. Paragraph counts came back identical in all three (345, 403 and 405 respectively). Heading-styled paragraphs across the set totalled 152 going in and 152 coming out. The single table in the corpus was unchanged at 7 rows and 14 cells.

Element: paragraphs, heading styles, table rows and cells. Layer: DOCX. Denominator: stated. Counting method: a script that reads the XML rather than asking the product what it did.

And the part you cannot check: the files are ours and we are not publishing them, so nobody outside this company can arrive at 152 by recounting. That is a real weakness of the number and I would rather name it than dress it up. What does transfer is the procedure — the six steps further down, run on a document you own instead of one we picked.

And now the parts a marketing page would leave out.

The wording of a heading is not covered by that number. On the 46-heading document, 18 headings came back reworded under the balanced setting while still carrying the same style. Surviving as a heading and surviving unchanged are two different claims, and only the first one is measured above.

Several elements were never tested, because the corpus did not contain any. Searching the packages for the markers of a TOC field, a cross-reference and a footnote returns nothing in all three files. That makes our count silent on exactly the elements a thesis depends on most. "The table survived" is a measurement. "Cross-references survive" would be an inference, and you should apply that distinction to anyone's version of the sentence, ours included.

Here is a failure we caught in our own research, which is why I trust the four questions more than I trust anyone's summary.

An internal write-up of that public dataset carried a line saying four named tools had scored 0% on heading survival. Recounting from the raw data, the list was right but it belonged to the bold column. On headings, one of those four companies is at 58.7%. Had that gone out, we would have published a 0% about a company whose public data says otherwise, and anyone could have shown it in five minutes by downloading the same file.

The number was real. It had simply become detached from what it measured, one document away from the source.

Which gives you a fifth question, and it is the cheapest one to ask: can I get back to the raw data from here, and who produced it? If the answer is a blog post citing a blog post, the number has already travelled further than it can safely go. If the raw data is one click away but the people who published it sell a tool in the same table, you can still use it — you just have to say both things at once, every time.

You do not need a benchmark to check a tool you are considering. You need one deliberately awkward file.

1. Build a test document with one of everything: two heading levels, an automatic table of contents, a cross-reference to a numbered figure, a footnote, a numbered list, a two-column table with a number and a unit in one cell, a quoted sentence, and three in-text citations from your reference manager.
2. Write down the counts before you upload. References, footnotes, table rows and cells, headings, list items.
3. Run it through the tool.
4. Open the result, select everything, and refresh the fields (Ctrl+A, then F9). A live field updates in front of you. A field that was turned into ordinary characters looks identical and does nothing, and this is the two-second way to tell them apart without opening the XML.
5. Recount, then compare the wording separately from the structure. A heading that is still a heading but says something different is a result, not a failure, but you need to know which one you got.
6. Read the quoted sentence and the cell with the number in it. Those two are where a fluent rewrite does the damage you cannot see by skimming.

Half an hour of this tells you more about your own file than any comparison page in this category can, ours included. Comparison pages, ours as much as anyone's, are written about somebody else's documents.

HumanPen is a document tool, so the file layer is the one we care about, and the counts above are what we have measured on it. Four elements remain unmeasured by us, and they are named in this article rather than left out of it.

The part that is a design decision rather than a measurement: nothing smaller than a whole paragraph is ever rewritten, and a selection landing in the middle of one is widened to that paragraph and put in front of you to confirm first. An imported report can set that boundary instead of you. Billing is drawn from the rewritten words themselves, which is a pricing fact, not a preservation claim, and I am listing it here only because the two get sold as one sentence. Our own FAQ still ends with "Review complex documents after download", which is the right instruction and not one we plan to soften.

The claim is narrow on purpose: fewer changed paragraphs means fewer places for any of this to go wrong.

Is there any independent benchmark for this? Not one that I can find. There is one open dataset, HumanizerBench, published under CC BY 4.0 — but it is operated by WriteHuman, which sells one of the tools it ranks, so independent is the wrong word for it. It is checkable rather than independent, and those are different properties: the raw files are downloadable, which is more than most of this category offers. It also measures Markdown formatting in plain text rather than anything inside a `.docx`. For the document layer I could not find a public dataset at all, which is why the do-it-yourself file above is worth the half hour.

A tool says it has a 99% preservation rate. Is that good? Unanswerable as written. Ask which element, in which file layer, out of how many, and counted how. A rate with no denominator and no named element is not a measurement.

What is the single fastest check? Ctrl+A then F9 in Word. It separates live fields from text that merely looks like them, and it takes two seconds.

Why do headings survive but reference lists sometimes not? Different mechanisms. A heading is a style attached to a paragraph, which survives a rewrite of the words inside it. A reference list is content, so whether it changes depends entirely on whether it was inside the scope of the rewrite in the first place.