How Uber Protects Against Retry Storms
4 hours ago
- Retry storms can cause significant business and brand damage, and existing mitigation strategies lack cross-service visibility and context-awareness.
- Naive retries amplify failures in deep dependency chains; for example, one retry per hop can exponentially increase request load on failing services.
- Retry budgets help control load but are manually configured and can reduce perceived availability when errors are not independent, such as during overload.
- Uber's solution introduces 'error ownership'—errors are attributed to the service that causes them, distinguishing between 'cause' errors and 'symptom' errors propagated from downstreams.
- The retry middleware uses error ownership to only retry requests for services that truly own the error, preventing exponential amplification while still allowing necessary retries.
- Service Dependency Analysis correlates inbound and outbound failures to establish error ownership, even in complex scenarios like coincidental errors.
- An 'at-least-once-retry' guarantee ensures availability is not harmed in situations where retries would help, while still preventing storms.
- In production, during a real outage, error ownership automatically stopped millions of spurious requests, reducing the maximum retry storm radius from 25 to 3 and the average from 20 to 2.
- The system is embedded in Uber's shared service mesh infrastructure, providing automatic protection to all services without custom code.