Training a model to identify AI-generated web content from structure alone
8 hours ago
- Word-level detectors fail under rewording and don't characterize text or identify the AI model.
- Proposes detection using structural signatures: how information is presented, order, evidence, and voice.
- Replicates StoryScope on commercial content with 2,250 human posts and 11,250 AI mirrors from five models.
- A 214-feature instrument, applied by an LLM and validated by humans, detects AI posts from structural features alone at 98.0 macro-F1 on held-out companies.
- Performance remains at 98.1 macro-F1 even when every AI post is reworded by its own model.
- AI posts share a tidy shape; 79.3% are attributed correctly to the source model against 16.7% chance, while human posts occupy rare configurations.
- All effects replicate StoryScope's, consistent in direction and larger in magnitude.
- The pipeline, instrument, prompts, code, and aggregate artifacts are released.