2 months ago
- A flaky node in a k3s cluster caused a cascade of failures, traced to slow SSD sync writes.
- Consumer SSDs' marketed IOPS (at QD32, no sync) don't reflect real-world sync workloads like etcd's 4K writes with fsync.
- The WD Red SA500 in a Proxmox VM performed poorly on QD1 fsync (134 IOPS), below etcd's 500+ IOPS requirement, causing cluster-wide issues.
- Slow disks compound through replicated storage (Longhorn), where one slow replica drags down the whole volume's latency.
- Power-loss protection (PLP) is key for enterprise SSDs to achieve high fsync throughput by caching writes safely.
- Kubernetes and etcd don't directly flag disk slowness; you must check kernel logs, load average, and iostat.
- fsck on corrupt volumes is a symptom fix; re-corruption indicates an ongoing storage performance problem.
- Virtualization (VM with virtio + LVM-thin) adds an IOPS tax, halving QD1 fsync performance compared to bare metal.
- The fix was to stop using the slow SSD for sync-heavy workloads, which resolved all upstream errors.
- Homelab experimentation builds debugging intuition and practical knowledge, valuable in production environments.