When "no healthy upstream" isn't about the upstream you think
2 days ago
- The incident involved intermittent failures in a search backend triggered by a slow dependency, not CPU throttling.
- The false root cause (CPU throttling) was initially appealing but contradicted by topology, timing, and failure distribution evidence.
- The actual cause was a combination of no timeouts on downstream calls and eager retries, creating a feedback loop that saturated the worker pool.
- Little's Law explains how increased request time (W) exponentially raises concurrency (L), overwhelming a fixed worker pool.
- The fix required setting explicit timeouts on downstream calls, bounding retries, and avoiding retries on connection timeouts.
- Failing fast keeps instances responsive and prevents cascading failures; degraded components should fail independently.
- Validation included load testing to ensure the fix works under both healthy and unhealthy dependency conditions.
- The key lesson is to treat root causes as hypotheses for verification, not conclusions, and to let evidence override elegant theories.
- Unbounded downstream calls and eager retries are latent outage risks that can turn a minor blip into a fleet-wide event.
- Diagnosis should focus on checking predictions against empirical evidence (topology, timing, distribution) rather than relying on plausible stories.