LLM Classification Is Feature Engineering
3 hours ago
- LLMs-as-classifiers are often poorly calibrated, making precision/recall trade-offs hard, and they don't reliably use all available information or adapt to different baselines.
- Instead of using LLM verdicts as final labels, they should be treated as features in a standard ML model like logistic regression, which improves calibration and allows flexibility.
- The feature engineering approach automatically calibrates predictions, incorporates structured features, and adapts to different populations through training data; it also improves interpretability.
- Challenges with prompt engineering can be addressed by treating prompts as learnable features: using training data, screening features, adding features from LLM subverdicts or log probabilities, and swapping model architectures.
- Case study on irony detection (SemEval 2018) shows that a simple LLM-based classifier outperforms the competition winner (0.747 F1), and feature engineering improves it further, achieving state-of-the-art results with a logistic regression on LLM features.
- There is growing research on using LLMs for feature extraction and automated feature engineering, as well as potential for agentic classifiers that use features of the investigative process itself.