24 days ago
- Database CPU spiked to 100% system time, causing service outages, with no obvious cause like new deploys or AWS incidents.
- Attempted database reboot via RDS failed for an hour, leading to alternative mitigation by blocking instance connections via security groups.
- After connections drained, database recovered but faced high user CPU time; resolved by identifying and removing expensive queries.
- Root cause was a lock convoy: 15k processes waiting on a GIN index extension due to synchronized bot events at peak times, triggering deadlock checks and log explosions.