Show HN: Calculate confidence score for OpenAI JSON output
a year ago
- #OpenAI
- #Confidence-Scoring
- #JSON
- Calculate confidence scores for OpenAI JSON outputs using granular field-level scoring.
- Works with OpenAI models supporting logprobs and structured JSON output (e.g., gpt-3.5-turbo, gpt-4).
- Simple integration with OpenAI's completion and chat endpoints.
- Lightweight package with zero dependencies for scoring logic.
- Example usage: Install via npm (@promptrepo/score and openai).
- API call setup with response_format and logprobs enabled.
- Parses JSON output and calculates confidence scores for each field.
- Supports nested JSON structures with detailed confidence scoring.