We reduced a container image from 800GB to 2GB

6 months ago

Sealos team reduced a container image from 800GB to 2GB, achieving a 99.7% reduction.
Root cause was a brute-force attack causing /var/log/btmp to grow to 11GB, compounded by OverlayFS's Copy-on-Write mechanism.
Developed a custom tool 'image-manip' to surgically remove files and squash 272 layers into one.
Implemented dedicated high-performance nodes for processing to handle the I/O load.
Resulted in 100% reduction in disk space alerts, 78% decrease in I/O, and $450/month savings.
Lessons learned include the need for automated monitoring and better base image configurations.

Hasty Briefsbeta