Softmax
Turns a vector of scores into a probability distribution that sums to 1.
Formula
Variables
Example
z=[1,2,3]: softmax=[0.09,0.24,0.67]
How it works
Softmax converts a list of raw scores into numbers that can be interpreted as probabilities. It does this by exponentiating each score, which makes larger scores stand out more, and then dividing by the total so the outputs add up to 1. The result is a smooth way to compare classes while preserving their relative strengths.
When to use it
Use softmax at the output of a multi-class classifier when each input should belong to exactly one class, such as digit recognition, topic classification, or choosing the next word in a language model. It is also useful whenever you want to turn arbitrary scores into a normalized probability distribution.
Worked example, step by step
Suppose the scores are z = [0, 1, 2]. Step 1: Exponentiate each score: e^0 = 1, e^1 ≈ 2.718, e^2 ≈ 7.389. Step 2: Add them up: 1 + 2.718 + 7.389 ≈ 11.107. Step 3: Divide each exponentiated score by the total: σ(z)_1 = 1 / 11.107 ≈ 0.090 σ(z)_2 = 2.718 / 11.107 ≈ 0.245 σ(z)_3 = 7.389 / 11.107 ≈ 0.665 Final softmax output: [0.090, 0.245, 0.665], which sums to 1.
Common mistakes to avoid
- Forgetting that softmax uses exponentials before normalizing
- Applying softmax to a single value instead of a set of class scores
- Assuming the largest score becomes 1 rather than just the largest probability
Did You Know?
Softmax is the standard output layer for multi-class classifiers, from digit recognition to LLMs.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Binary Cross-Entropy
BasicLoss for binary classification comparing predicted probability to the true label.