Scanning for Pangram Errors
6 days ago
- Pangram is an AI detection tool with a claimed false positive rate of 1 in 10,000.
- The author tested Pangram on 2.9 million words from 45 old books that were never digitized, to check for memorization of training data.
- Pangram did not flag any human-written content as AI (false positives) except for segments where OCR (Mistral) hallucinated text, which Pangram correctly identified as AI-generated.
- The results suggest Pangram is not simply memorizing its training set but is detecting AI writing patterns.
- Human review is recommended when using Pangram for important decisions, as the tool itself is not infallible.