The Normalization of Inexplicable Failures
2 hours ago
- The author compares inexplicable software failures to broken doors with obstructions, emphasizing that software should not 'suck' without a clear reason.
- Slack is cited as an example of a widely adopted product that fails to reliably deliver messages, highlighting a disconnect between adoption and reliability.
- Jev, an AI model that returns typed values with probabilities, is fast and cheap, but using it properly still requires building evals and ground-truth pipelines—the hard part.
- Confidence scores from AI models are often poorly calibrated and misused, leading to arbitrary thresholds or 'cargo cult' behavior instead of informed decision-making.
- Traditional software errors have defined accountability, but LLM-driven failures often lack concrete causes, normalizing the acceptance of 'sometimes it just sucks.'
- The author fears that normalizing inexplicability prevents debugging and improvement, even though LLM-accelerated development could help if we invested in proper evaluation.