Machine Learning & AI

Advanced

KL Divergence

Measures how one probability distribution differs from a reference distribution.

Formula

D_{KL}(P\Vert Q)=\sum_i P(i)\log\dfrac{P(i)}{Q(i)}

Variables

PTrue distribution

QApproximating distribution

Example

Zero only when P equals Q

Did You Know?

KL divergence is asymmetric — the distance from P to Q is not the same as Q to P.

information theory loss vae

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.