A Weekend TF-IDF + SVM Classifier Spots LLM Text at 85% Per Sentence

A developer set out to build an AI-text detector after growing frustrated with low-quality, obviously machine-written fanfiction flooding online tags. Perplexity-based approaches — scoring how predictable each word is under a reference LLM — proved unreliable, plagued by false positives, high inference cost, and poor generalization across models. Instead, the author fell back on classical machine learning: scikit-learn’s TF-IDF features feeding a LinearSVC classifier. The bet was that current LLMs have distinctive, statistically detectable word-choice patterns, and the signal turned out to be far stronger than expected.

Training data came from roughly 10,000 pre-ChatGPT (2010–2022) human-written texts paired with an equal set of AI regenerations, produced by summarizing the originals and having seven different models (Gemini 3 Pro, Qwen, GLM-5, GLM-4.7, Kimi, Doubao, DeepSeek) rewrite them — with API costs kept near zero by exploiting free and promotional channels, which the author concedes violates platform terms. Splitting text into sentences and classifying each hit around 85% accuracy with over 80% F1. Because errors wash out over many sentences, whole articles can be judged with high confidence. A single 8-class model failed (~50%, likely because the models are distilled from one another), so the author trained seven binary classifiers and flagged a sentence as AI when at least two agreed.

The author suggests this is likely how many commercial “AI plagiarism checkers” actually operate under the hood, and argues the lightweight classical pipeline outperforms tools that simply ask an LLM whether text is machine-generated. Notably, Claude’s naive attempt to dump raw text into a model produced a suspicious 99.45% accuracy (a data-leak red flag), and later went off-script by reimplementing TF-IDF and SVM in JavaScript for a serverless web demo. Code and trained models are published on GitHub, with a browser-based demo available, though the author stresses the model is unoptimized and not trained for general-purpose use.