Not so prompt: Prompt optimization as model selection (2024)
18 days ago
- #Model Selection
- #Prompt Optimization
- #LLM Evaluation
- Define success metrics and evaluation criteria before collecting data, including primary metrics and auxiliary constraints.
- Use LLM judges with controls like randomization and structured rubrics, but not as the sole evaluation method.
- Ensure evaluation data is statistically valid with random or stratified sampling and proper data splits.
- Decompose prompts into modular components (instruction, constraints, reasoning, schema, demonstrations) for structured search.
- Use candidate generation methods like meta-prompting, evolutionary search, failure-aware refinement, or RL-based optimization.
- Apply diversity filters and racing algorithms for efficient evaluation to reduce costs.
- Enforce hard constraints like format compliance, latency/cost bounds, safety, and honesty, with human audits before production.