Detecting LLM Text with Classical ML: TF-IDF Still Works

A developer has published a detailed writeup on building an LLM text detector using classical machine learning techniques - specifically TF-IDF vectorization with linear Support Vector Classifiers. The approach achieved approximately 85% accuracy at the sentence level, demonstrating that you do not need deep learning to detect AI-generated content. But as the HN discussion reveals, whether this matters depends entirely on your use case.

The Technical Approach#

The system uses a straightforward pipeline: TF-IDF feature extraction feeding into seven binary SVM classifiers, one per LLM model family (Doubao, Qwen, GLM-5, Kimi, Deepseek, and others). Even a "buggy first version" hit 88% accuracy according to the author.

Key metrics:

• Doubao classifier: 89.4% accuracy, 87.0% F1 score
• Qwen classifier: 89.1% accuracy, 89.7% F1 score
• Average across models: 85.9% accuracy
• Web deployment version (500k features): ~84% accuracy

The training dataset combined approximately 10,000 human-written texts from 2010-2022 web fiction platforms with matching LLM-generated counterparts, totaling over 8.5 million sentences.

A majority-voting ensemble flags sentences as AI-generated when two or more of the seven classifiers trigger. Testing against pre-2022 literature showed false positive rates below 0.5% at a 50% detection threshold.

Bypass attempts largely failed:

• Translation roundtrip (Google/Youdao): 89.9% to 79-85% accuracy
• LLM prompt rewriting: 89.9% to 79-83% accuracy

The detector still caught most attempts even after deliberate evasion.

What HN is Saying#

The discussion surfaced the fundamental tension in AI detection: accuracy versus consequences.

The Arms Race Argument: Multiple commenters argued that detection is inherently temporary. "In training all you have to do is take their model as the adversary and then it's useless," one noted, referencing GAN-style adversarial training. The spam filtering analogy came up repeatedly - Bayesian filters worked great until spammers adapted.

The False Positive Problem: "Imagine how soul-crushing writing an entire dissertation by hand and having it rejected because some 'good enough' LLM detector decides you write too much like an AI." This concern resonates particularly in educational contexts where false positives can end academic careers.

A commenter recalled a 2023 incident where a Texas professor used an anti-plagiarism tool that flagged over one-third of a class as AI users - and the professor's own decade-old published work also got flagged when students tested it.

The Language Drift Question: Several commenters pointed out that LLMs are changing how humans write. "Generation alpha is going to have a lot of trouble if we keep perpetuating the myth that you can really interpret text in an ongoing fashion." As humans absorb LLM-influenced text, the baseline for "human writing" shifts.

The Practical Defense: Others argued that most AI slop is low-effort. "The thing about most text slop is how little effort goes into disguising it. If you can catch some of it, that's something at least." Commercial chat models are specifically tuned for engagement in ways that create detectable patterns - users who want to evade detection can, but most do not bother.

The Pangram Comparison: Users familiar with existing detection tools noted that Pangram claims a 1-in-10,000 false positive rate, tested against pre-2020 texts. The post's approach independently discovered a core technique Pangram uses - creating "twins" to compare human and AI text distributions.

The Technical Debate#

Why classical ML works here:

Commercial LLMs are optimized for engagement through RLHF, not for evading classifiers. This creates consistent stylistic patterns - em-dash overuse, specific sentence structures, particular phrasing habits. These patterns are exactly what TF-IDF captures well.

One commenter observed: "You don't need a style model - current models are very good at doing 'style transfer' of a model text onto whatever it has written if you just have it do it chunk by chunk." The counter-argument: most users generating AI slop are not doing this extra step.

The scalability question:

Could a detector run as a browser extension against every paragraph displayed? The classifier is small enough that this seems feasible. One commenter proposed "an anti-slop blocker" analogous to ad blockers - not catching everything, but filtering the low-effort cases.

The provenance question:

"Whether a text was written by a human or not is just a single bit of information. So you can't rule out its detectability a priori, since even the shortest text contains more information than that."

This got pushback: for any given text, both humans and machines could have written it. The data is fundamentally inseparable for many cases - there is no unique provenance label for each possible string.

What This Means for Developers#

If you need to detect AI text in bulk - screening content submissions, filtering training data, moderating forums - classical ML approaches offer a reasonable starting point without the complexity of neural networks.

The 85% sentence-level accuracy drops with shorter text. The false positive rate below 0.5% on pre-2022 text is encouraging, but language drift means this will degrade over time without retraining.

For high-stakes decisions (academic integrity, hiring, legal), no automated detector should be trusted without human review. The false positive risk is too high and the consequences too severe.

The arms race framing is probably correct at the limit - a sufficiently motivated adversary can evade any classifier. But most AI text comes from people using commercial models without any evasion effort. For that common case, classical ML detection works well enough to be useful.

The Uncomfortable Truth#

The HN discussion kept circling back to a meta-question: why do we care?

If the text is useful and accurate, does the provenance matter? If a forum post answers a technical question correctly, does it matter that GPT-5 wrote it?

The answer depends on context. Academic work requires demonstrating your own understanding. Creative writing presumably values human expression. Technical documentation mostly cares about accuracy.

For many use cases, the real question is not "was this written by AI" but "is this content good." Detection tools answer the wrong question - but for contexts where provenance genuinely matters, the classical ML approach here provides a surprisingly effective baseline.

Sources#

• Original Blog Post
• Hacker News Discussion
• Pangram Research Paper
• Wikipedia: Signs of AI Writing

FAQ#

Can LLM text detection work reliably?#

For low-effort AI content using commercial models, classical ML approaches achieve 85%+ accuracy. High-stakes decisions still require human review due to false positive risk.

Why does TF-IDF work for detecting LLM text?#

Commercial LLMs are optimized for engagement through RLHF, creating consistent stylistic patterns in word choice and sentence structure that TF-IDF captures effectively.

Can users bypass LLM text detectors?#

Yes - translation roundtrips and prompt rewriting reduce detection accuracy by roughly 10 percentage points. Sufficiently motivated users can evade detection, but most do not bother.

Should schools use AI text detectors?#

Caution is warranted. False positives can have severe consequences, and some students' natural writing style may resemble LLM output. Human review should always accompany automated detection.