A $196 fine-tuned 7B model outperforms OpenAI o3 on document extraction
6 hours ago
- #language-model
- #machine-learning
- #information-extraction
- Extract-0 is a 7-billion parameter language model optimized for document information extraction.
- It outperforms larger models like GPT-4.1, o3, and GPT-4.1-2025 on a benchmark of 1,000 tasks.
- Training involves synthetic data generation, supervised fine-tuning with LoRA, and reinforcement learning via GRPO.
- The model modifies only 0.53% of weights (40.4M out of 7.66B parameters) for efficiency.
- A novel semantic similarity-based reward function handles ambiguity in extraction tasks.
- Demonstrates task-specific optimization can surpass general-purpose models with fewer resources.