- Keygen experienced two outages on February 5 and February 6, 2024, each lasting over five hours, due to a customer's sudden traffic spike and flawed retry logic causing an unintentional DDoS.
- The root cause was performance issues with the sidekiq-unique-jobs plugin, which iterated large Redis sorted sets synchronously (using Lua), blocking Redis and causing timeouts and slow response times.
- Cloudflare's CDN modified the Date header on API responses, breaking response signature verification when the Orange Cloud was enabled, forcing a trade-off between protection and compatibility.
- The author implemented an application-side rate limiter to replace Cloudflare's, and changed heartbeat job scheduling from future-dated jobs to a cron-based cleanup to avoid large queues.
- After clearing the bloated schedule queue and removing the unique-jobs bottleneck, overall API response times dramatically improved.
- Remediations include adopting Fastly's Next-Gen WAF, replacing sidekiq-unique-jobs with Sidekiq Pro/Enterprise (opt-in for self-hosted), and improving alerting and paging to bypass DnD mode.