KL Divergence
Measures how one probability distribution differs from a reference distribution.
Formula
Variables
Example
Zero only when P equals Q
How it works
KL divergence measures the expected extra information, in nats or bits depending on the log base, that you pay when you treat samples from P as if they came from Q. It comes from comparing the log-likelihoods assigned by the two distributions: if Q assigns lower probability to outcomes that P often produces, the penalty grows quickly. Because the weighting uses P, the divergence is asymmetric and is best thought of as a directed mismatch from the true distribution toward the model or approximation.
When to use it
Use KL divergence when you want to compare a target distribution with a model output, such as in variational inference, language modeling, Bayesian updating, or measuring how much one classifier's predicted probabilities differ from a reference. It is especially useful when you care about probability calibration and not just the most likely class.
Worked example, step by step
Let P = [0.5, 0.3, 0.2] and Q = [0.4, 0.4, 0.2]. Use natural log, so the result is in nats. D_KL(P||Q) = 0.5 ln(0.5/0.4) + 0.3 ln(0.3/0.4) + 0.2 ln(0.2/0.2) = 0.5 ln(1.25) + 0.3 ln(0.75) + 0.2 ln(1) = 0.5(0.2231) + 0.3(-0.2877) + 0.2(0) = 0.1116 - 0.0863 + 0 = 0.0253 nats So the KL divergence from P to Q is approximately 0.025 nats.
Common mistakes to avoid
- Thinking KL divergence is symmetric; in general D_KL(P||Q) != D_KL(Q||P)
- Using probabilities of zero in Q without checking support, which can make the divergence infinite if P(i) > 0
- Forgetting that the log base changes the units: natural log gives nats, base 2 gives bits
Did You Know?
KL divergence is asymmetric — the distance from P to Q is not the same as Q to P.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.