- The paper proposes a theory explaining why contrastive learning with simple images and augmentations yields useful representations for downstream tasks.
- It analytically computes the optimal representation for a contrastive loss given basic augmentations and any image dataset with stationary statistics.
- For certain augmentations, the optimum can be achieved by a CNN with sinusoidal first-layer filters, a pointwise nonlinearity, global average pooling, and a final linear layer performing partial whitening.
- Even with more complex augmentations, the optimal weights in such CNNs remain sinusoidal, with frequencies and weights computable via a waterfilling algorithm based on the dataset's expected power spectrum.
- Experiments with various image datasets and augmentations confirm that CNNs trained with SGD empirically learn sinusoidal filters and partial whitening, aligning with the theoretical predictions.