Tanh Activation
Squashes input to the range (-1, 1); zero-centred activation.
Formula
Variables
Example
tanh(0)=0, tanh(2)=0.964
How it works
The tanh activation converts any real input into a value between -1 and 1 by comparing the size of the positive and negative exponentials. When x is large and positive, e^x dominates so the output approaches 1; when x is large and negative, e^-x dominates so the output approaches -1. Because it is centred around 0, it often helps neural-network layers produce outputs with both positive and negative values, which can make learning easier than using a strictly positive activation.
When to use it
Use tanh in hidden layers when you want outputs that are smooth, bounded, and zero-centred, especially in smaller or older neural-network architectures. It is also useful in recurrent models and other settings where keeping activations within a limited range helps control unstable values.
Worked example, step by step
Take x = 1 Step 1: Substitute into the formula: tanh(1) = (e^1 - e^-1) / (e^1 + e^-1) Step 2: Evaluate the exponentials: e^1 ≈ 2.718 and e^-1 ≈ 0.368 Step 3: Compute the numerator: 2.718 - 0.368 = 2.350 Step 4: Compute the denominator: 2.718 + 0.368 = 3.086 Step 5: Divide: tanh(1) ≈ 2.350 / 3.086 ≈ 0.762 Final answer: tanh(1) ≈ 0.762
Common mistakes to avoid
- Thinking tanh can output values outside the interval (-1, 1)
- Forgetting that tanh is zero-centred, so tanh(0) = 0
- Mixing up tanh with sigmoid, which outputs values from 0 to 1
Did You Know?
Because tanh is zero-centred it often trained early networks better than the sigmoid.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.
Binary Cross-Entropy
BasicLoss for binary classification comparing predicted probability to the true label.