Triage GitHub Pull Requests with OpenAI's Decisions API
3 hours ago
- OpenAI's Decisions API evaluates a GitHub pull request's diff against specific review questions, such as identifying which review track a change belongs to or whether existing callers need migration guidance.
- Effective review questions focus on narrow, actionable concerns (e.g., 'Does this change require callers to update their code?') rather than broad judgments like 'Is this PR good?'
- Evidence provided to the API should include the PR title, description, changed file paths, and patch text, but diffs have limitations—additional context like interface documentation may be needed.
- The API returns predictable answer types (choice, predicate, refusal) which application code can use to organize review, but results must be tied to specific commits and rerun if the PR changes.
- PR triage does not verify correctness or readiness to merge; separate tests, reviews, and branch protections remain necessary, and the model's output should be validated against reviewer feedback.
- The example Node.js script collects PR data from GitHub's REST API and sends it to OpenAI, handling constraints like a 100-file limit and 100 KB input budget.
- The API does not fetch PR data itself—your application must retrieve and supply evidence, and it can handle multiple independent questions in one request.