Demystifying Noise Contrastive Estimation
2 days ago
- #statistical-estimation
- #machine-learning
- #contrastive-learning
- Noise Contrastive Estimation (NCE) and InfoNCE are methods for estimating statistical distributions by distinguishing real data from noise.
- Local NCE (Binary NCE) reduces learning p(x|c) to binary classification, using self-normalization and noise sampling to approximate the partition function.
- Global NCE (Ranking NCE) uses a categorical classifier to identify the real sample among multiple candidates, learning a density ratio p(x|c)/q(x).
- InfoNCE maximizes mutual information I(x;c) by learning the density ratio without knowing q(x), underlying contrastive learning techniques like CLIP and SimCLR.
- Applications include language modeling, speech recognition, reinforcement learning, computer vision, and GANs, with differences in assumptions, scoring functions, and normalization.