Softmax Function
Converts a vector of scores into a probability distribution.
Formula
Variables
Example
Scores [1,2,3] give ~[0.09, 0.24, 0.67]
How it works
The softmax function turns a set of raw class scores into positive numbers that add up to 1, so they can be interpreted as probabilities. It does this by exponentiating each score, which keeps the ordering of the scores but amplifies relative differences, and then normalizing by the total over all classes. In practice, the class with the largest score gets the largest probability, but every class still receives some nonzero mass unless its score is far smaller than the others.
When to use it
Use softmax at the output of multiclass classification models when you need a probability distribution over mutually exclusive classes, such as image labels, intent classification, or next-token prediction. It is also useful when comparing alternatives probabilistically, because the outputs are directly interpretable and can be trained with cross-entropy loss.
Worked example, step by step
Suppose the model outputs three scores: z = [0.5, 1.5, -0.5]. Step 1: Exponentiate each score: e^0.5 ≈ 1.6487, e^1.5 ≈ 4.4817, e^-0.5 ≈ 0.6065 Step 2: Sum the exponentials: 1.6487 + 4.4817 + 0.6065 = 6.7369 Step 3: Divide each exponential by the total: σ(z1) = 1.6487 / 6.7369 ≈ 0.2447 σ(z2) = 4.4817 / 6.7369 ≈ 0.6652 σ(z3) = 0.6065 / 6.7369 ≈ 0.0900 Final probabilities: [0.2447, 0.6652, 0.0900]
Common mistakes to avoid
- Applying softmax to a single scalar instead of a vector of class scores
- Forgetting that the outputs must sum to 1 after normalization
- Computing exp(z_i) directly on very large scores without using the numerically stable shifted-softmax trick
Did You Know?
Softmax outputs always sum to 1, making them interpretable as probabilities.
Share this formula
More in Data Science & ML
View allMean Squared Error (MSE)
BasicAverage of squared differences between predicted and actual values.
Root Mean Squared Error (RMSE)
BasicSquare root of MSE, in the same units as the target variable.
Mean Absolute Error (MAE)
BasicAverage of absolute differences between predictions and actual values.
Coefficient of Determination (R²)
IntermediateProportion of variance in the target explained by the model.
Classification Accuracy
BasicFraction of predictions that are correct.
Precision
IntermediateOf all positive predictions, the fraction that were correct.