Cross-Entropy and KL Divergence
a year ago
- #information-theory
- #probability
- #machine-learning
- Cross-entropy is used in ML for classification loss computation.
- Information content of an event is defined using logarithms, with base 2 for bits.
- Entropy measures uncertainty in a random variable, with high entropy indicating high uncertainty.
- Cross-entropy extends entropy to compare two probability distributions, P (actual) and Q (predicted).
- KL divergence adjusts cross-entropy by subtracting entropy of P, providing a divergence measure.
- In ML, cross-entropy serves as a loss function, optimizing it is equivalent to optimizing KL divergence.
- Maximum Likelihood Estimation relates to cross-entropy minimization, linking statistical estimation to ML loss functions.