Categorical Cross-Entropy
Multi-class loss summing the log-probability of the correct class.
Formula
Variables
Example
true=class2, p=0.7: L=-log0.7=0.357
How it works
Categorical cross-entropy measures how well a model's predicted probability distribution matches the true class. It gives a small loss when the model assigns high probability to the correct class, and a large loss when it assigns low probability, because the logarithm heavily penalizes confident wrong predictions. With one-hot labels, only the true class contributes to the sum, so this formula is essentially the negative log-probability of the correct answer.
When to use it
Use this loss when a model must choose exactly one class from several options, such as image classification, spam detection, or topic prediction. It is especially common with a softmax output layer, where the model produces probabilities for each class.
Worked example, step by step
Suppose there are 3 classes and the true label is class 1, so y = [1, 0, 0]. The model predicts probabilities \hat{y} = [0.2, 0.5, 0.3]. Apply the formula: L = -\sum_i y_i \log(\hat{y}_i) L = -(1\cdot\log(0.2) + 0\cdot\log(0.5) + 0\cdot\log(0.3)) L = -\log(0.2) L \approx -(-1.609) L \approx 1.609
Common mistakes to avoid
- Using the predicted class label instead of the full probability vector
- Forgetting that one-hot labels make all non-true classes contribute 0
- Taking the log of 0 or using raw scores instead of probabilities
Did You Know?
Paired with softmax, this loss has a beautifully simple gradient: predicted minus true.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.