Notes on incidents
2 days ago
- Most incidents involve waiting and often resolve on their own without intervention.
- Engineers often make incidents worse by jumping to quick fixes, like clearing queues or forcing redeploys.
- The first recommended action during an incident is to pause and relax, using a ritual to avoid rushing.
- Effective incident resolution actions are usually simple, such as temporarily disabling a problematic feature.
- Knowledge of the system is more valuable than having many engineers on the call.
- Resolving an incident requires courage and decisiveness, especially from those with system knowledge.
- Fixing incidents earns political credit with managers and executives, but this is not a durable position of power.
- The tendency for incidents to self-resolve may depend on system size, with larger systems having more self-recovery mechanisms.