Training a 4B model to produce 81% faster query plans than Postgres
3 hours ago
- Query optimizers are notoriously difficult, with join ordering being NP-hard, leading to suboptimal plans even after decades of research.
- The goal is to train a small language model (4B parameters) to generate pg_hint_plan hints that produce faster PostgreSQL query plans than the default optimizer.
- The training pipeline uses off-policy distillation from GPT-6 Astra trajectories for supervised fine-tuning (SFT), followed by agentic reinforcement learning (GRPO) with custom rewards.
- Measurement noise in query execution times was mitigated by tuning shared_buffers to 2 GB and using a three-run interleaved median comparison with a 5% tie zone.
- The final 4B model achieved a 1.81x geometric mean speedup and a 44.7% total latency reduction across 113 join-heavy JOB queries, outperforming PostgreSQL's default plans.
- The project demonstrates that small, open-weights models can be effectively post-trained for niche domain tasks using RL, with a total cost of approximately $1,200.
- Key infrastructure included a rented 2x H100 node for training/inference and four local PostgreSQL containers for measurements, connected via Tailscale.
- The model learned to construct valid join trees and use the qo-agent harness, going from zero valid candidates in 99/113 queries to successful plans in 101/113 queries after RL.