svd

Deriving the SVD (Single Value Decomposition) from scratch

3 months ago

The author discusses the challenge of truly learning and grasping mathematical concepts, noting that traditional math books often present the final formalized version without showing the messy, experimental path that led there.
Linear Algebra is chosen as a starting point due to its applicability and accessibility, with a focus on deriving the Singular Value Decomposition (SVD) in an intuitive manner by connecting it to everyday intuition and problem-solving motivation.
The SVD is introduced as a way to extract structure from any linear transformation, analogous to diagonalization for nice matrices, by representing it as a product of orthonormal input and output bases and a diagonal matrix of singular values.
Key insights include the orthonormality of both input and output frames, the equality of ranks between a matrix A and AᵀA, and the geometric interpretation involving the four fundamental subspaces (row space, kernel, image, and orthogonal complement).
The SVD can be expressed as a sum of rank-1 outer products, which leads to applications in compression (via the Eckart–Young theorem) and Principal Component Analysis (PCA), where singular values rank the importance of directions in data.
The entropy of the singular-value distribution connects linear algebra to information theory, measuring how much information a matrix carries versus redundancy, and hinting at deeper concepts like Kolmogorov complexity for future exploration.