Postgres locks do not scale
a day ago
- #incident-postmortem
- #aws-rds
- #postgres
- Database CPU spiked to 100% system time, causing service outages, with no obvious cause like new deploys or AWS incidents.
- Attempted database reboot via RDS failed for an hour, leading to alternative mitigation by blocking instance connections via security groups.
- After connections drained, database recovered but faced high user CPU time; resolved by identifying and removing expensive queries.
- Root cause was a lock convoy: 15k processes waiting on a GIN index extension due to synchronized bot events at peak times, triggering deadlock checks and log explosions.
- Key lessons include monitoring wait queues, isolating GIN indexes, and designing for bursty workloads to prevent synchronized operations from causing system failures.