- Define quality measurement, set a baseline, and build an automated evaluation pipeline without relying on human judgment.
- Use offline evaluation with a static dataset and rubric-based scoring (e.g., LLM-as-judge) to test changes locally.
- Include safety-critical inputs like canary or honeypot prompts to catch regressions, evaluated with pass/fail annotations.
- Structure evaluation using traces, datasets, annotations, and experiments; store data for comparison and report generation.
- Start with a local pipeline to refine rubrics before automating or adopting tools like Langfuse or Opik for online evaluation.