9 hours ago
- Latency spikes during BGSAVE in Redis were not caused by disk I/O or the fork() call itself (fork took 6ms, spike ~300ms).
- Stack traces from Redis software watchdog showed the process was blocked on MOV operations after fork, indicating lazy page table copying by Linux.
- Linux optimizes fork() by not copying page table entries (PTEs) immediately; copy-on-write faults later cause delays, especially with many writes.
- The issue is not EC2-specific but more noticeable on virtualized instances due to slower PTE copying.
- Testing with jemalloc vs libc malloc revealed jemalloc caused higher latency spikes (339ms vs 21ms on bare metal) due to transparent huge pages (THP).
- Transparent huge pages used by jemalloc cause large copy-on-write workloads when the parent process touches shared pages after fork.
- The solution is to disable transparent huge pages using: echo never > /sys/kernel/mm/transparent_hugepage/enabled
- Building Redis with MALLOC=libc can reduce latency but may increase memory fragmentation; disabling THP is preferable.