Running Out of Disk Space in Production
4 days ago
- #nginx configuration
- #server administration
- #disk space management
- A server hosting Kanjideck files went live, causing immediate overload as hundreds of customers tried downloading a 2.2GB file.
- Disk space filled up, leading to errors and failed downloads, with root cause traced to nginx buffering large files to temporary storage.
- Temporary fixes included clearing logs, garbage collection, and moving the Nix store to a separate volume to free up space.
- Investigation revealed nginx's proxy_max_temp_file_size and proxy_buffering settings were causing disk usage spikes and download failures.
- Disabling buffering and setting proxy_max_temp_file_size to 0 resolved the issues, stabilizing disk usage and allowing successful file downloads.