Why “Undetectable” Claims Fall Apart Under Independent Testing
Ninety-eight percent. That was the accuracy figure Workado put on its AI content detector. When the US Federal Trade Commission checked, the tool managed 53% on general-purpose content, a hair better than a coin toss (FTC, April 2025). The company now sits under a final order to stop making claims it can’t support.
That case was about a detector, not a rewriting tool. Still, the pattern behind it runs all through the AI writing market, and it’s loudest among products promising to make AI text “undetectable.” Big round numbers on the homepage. Almost nothing about how anyone got them.
So why do these claims crumble the moment someone else runs the test? And what does a believable result actually look like?
The Gap Between a Homepage Number and a Measured One
Three claims, three very different results
In September 2026, Phrasly published its humanizer Benchmarks, a test of four rewriting tools against Pangram’s newest detection model, pangram-4. Phrasly sells one of those tools, so read it with that in mind. What makes it useful anyway is that it puts each competitor’s advertised claim right next to the measured result, and it publishes per-output score files anyone can download.
The contrasts are hard to miss:
- WriteHuman’s homepage cites a 94% Pangram human score with no date, sample size or method attached. Across 69 texts, its measured average was 13.3%.
- StealthGPT’s API page cites an 89% bypass rate against Pangram V4, drawn from an internal 100-sample test its founder described on September 8, 2026. In the 72-text run, 53% of its outputs reached even a 50% human score.
- Undetectable AI doesn’t advertise a Pangram figure at all. Its measured average came in at 24.5%.
Advertised figures versus measured results. Sources: FTC (April 2025); Phrasly humanizer benchmark (September 2026). The StealthGPT pair uses different pass definitions, so treat it as indicative.
Why vendor tests and outside tests disagree
Nobody has to be lying for gaps like these to open up. Most of the time, four things explain it.
The sample is small or hand-picked
A vendor can run a dozen texts, keep the ones that pass, and screenshot the winners. Even an honest internal test of 100 samples only tells you how the tool did on texts the company chose.
The detector has moved on
Detectors ship new models constantly. StealthGPT itself said Pangram 4 broke its earlier rewriting models, which is why it launched a new one called Super (RuntimeWire, September 2026). A “99% bypass” screenshot from last spring may describe a detector nobody runs anymore.
The metric is loosely defined
“Bypass rate” can mean scoring below 50% AI, scoring 100% human, or simply getting a “likely human” label. Same outputs, three different headline numbers. Pick the friendliest one and you’ve got a homepage stat.
Nobody checks the meaning
A rewrite that slips past a detector by inventing facts isn’t a win. StealthGPT’s own evaluation gave Super an 82% factual-consistency pass rate, so 18 of 100 outputs failed its check (RuntimeWire, 2026). That’s the vendor’s number, and to its credit, it published it. Most don’t.
What Independent Research Says About Humanized Text
Academic work hasn’t been kind to the “undetectable” pitch, at least against the strongest detectors. Economists Brian Jabarian and Alex Imas ran AI-written passages through StealthGPT and tested four detectors on the results (NBER working paper 34223, September 2025). GPTZero, they found, “largely loses its capacity” to catch the humanized text. Pangram barely flinched, with false negative rates topping out around 5%.
Pangram’s own August 2025 report on 20 humanizers points the same way, though it comes from a company with an obvious interest. By its count, the detector still caught 90.3% of Undetectable AI’s output and 95.6% of StealthGPT’s. Put that next to the bypass rates those tools advertised at the time and you get a familiar picture. Each side runs its own test, and each side wins it.
Why every company seems to win its own test
This is the part buyers rarely hear. A test is a set of choices: which texts, which detector, which settings, which threshold, which outputs count. Change any one and the result moves. A company running its own benchmark doesn’t have to fake anything. It only has to make each choice in its own favor, and most people won’t notice, because the choices never appear on the homepage.
That’s why a single number, from anyone, deserves less weight than a published method.
So two things hold at the same time. Humanizers can beat some detectors. They fail badly against others. A claim of “undetectable” that doesn’t name the detector, the version and the test date tells you close to nothing.
It also explains why regulators are paying attention. The Workado order wasn’t about rewriting tools, but it set out a simple standard: if you advertise an accuracy number for an AI product, you need evidence that supports it. There’s no reason to think bypass rates get a pass.
Red Flags in an “Undetectable” Claim
Before you trust a percentage on a pricing page, look for these:
- No sample size. If they won’t say how many texts, assume very few.
- No detector version or test date.
- Screenshots only, with no data you can inspect.
- One detector, chosen by the vendor.
- Silence on meaning and factual accuracy.
- Fine print that undercuts the headline. WriteHuman’s own disclaimer, quoted on the benchmark page, admits that no humanizer can guarantee a specific Pangram score.
That last one is more common than you’d expect. The bold claim lives at the top of the page. The honest sentence lives in the FAQ.
What a credible test looks like
A test worth trusting tends to share a few habits. Phrasly’s September run is a reasonable template, whatever you make of who ran it:
- Inputs come from a public dataset, not the vendor’s drawer. Here, 72 English texts of 604 to 900 words were sampled at random from the AI Text Detection Pile on Hugging Face.
- Each tool gets one output per text. No rerolling until something passes.
- Tool versions and settings are spelled out, down to Undetectable AI’s API model and strength setting.
- Exclusions are disclosed. WriteHuman declined 3 of the 72 texts, and those gaps are reported rather than quietly dropped.
- Results can be downloaded, and every output carries a SHA-256 fingerprint so it can’t be swapped later.
- Quality is measured next to detection. A hallucination count sits beside every score.
Even with all that, check whether the publisher has a stake. The same benchmark shows Phrasly Ultra with a 99.8% average human score, which is the headline. It also shows that 51.4% of Ultra’s outputs contained at least one hallucination. That second figure is exactly the kind of detail marketing usually trims, and its presence is a decent sign the test wasn’t written backwards from the conclusion.
For a smaller, hands-on comparison, Phrasly’s six-detector StealthGPT test shows how far one tool’s scores can swing from detector to detector. It comes from a competitor too, so weigh it the same way.
Trust the Method Before the Headline
“Undetectable” is a marketing word, not a measurement. The tools behind it may work well against one detector, poorly against another, and differently again next month after a model update. The only way to know is to look at how the number was produced.
So flip the habit. Skip the percentage and go straight to the method: how many texts, which detector, which version, what date, and what happened to the facts along the way. If a vendor can’t answer those questions, the number isn’t evidence. It’s decoration.
And if a test does answer them, including the parts that make its own product look less than perfect, that’s usually the one worth reading twice.