- Compression can be lossy (accepting errors for higher compression ratios) or lossless (reconstructing data exactly but with lower compression).
- Using a linear regression model on 1000 apartment price data points, compression ratios vary: 41:1 for lossy naive encoding, 21:1 for lossy entropy-based encoding, and 1.18:1 for lossless entropy-based encoding.
- The lossless compression includes the cost of residuals, which constitute most of the bits (94% in the example), showing that model benefits are modest in bit savings.
- Compression ratios depend on encoding methods and purpose: lossy compression suits storage savings when approximate data is acceptable, while lossless ensures exact reconstruction for future analysis.
- The asymptotic ceiling for lossless compression is determined by the inherent structure and noise in the data (a property of the world), not by the model, limiting compression gains.
- Residuals in lossless compression preserve information for future model improvements, acting as a 'to-do list' for uncovering additional data patterns.
- The audit method applies universally to models like clustering, decision trees, and neural networks, with training loss analogous to residual costs in compression.