Positional Encoding
Injects word-order information into a Transformer that otherwise ignores position.
Formula
Variables
Example
Different frequencies encode position uniquely
How it works
Transformers do not have any built-in sense of word order, so positional encoding adds a location-dependent signal to each token embedding. In this sinusoidal version, each embedding dimension uses a different wavelength, letting nearby positions change smoothly while distant positions produce distinct patterns. The mix of many frequencies makes each position identifiable and gives the model a way to infer relative offsets from the differences between sine and cosine components.
When to use it
Use this in sequence models such as language translation, text generation, or time-series Transformers when the architecture needs explicit information about order. It is especially useful when you want a fixed, non-learned positional scheme that can generalize to sequence lengths not seen during training.
Worked example, step by step
Take d = 8, pos = 5, and i = 1. Compute the exponent: 2i/d = 2(1)/8 = 0.25 Compute the denominator: 10000^0.25 = 10 Plug into the formula: PE_(5,2) = sin(5/10) So PE_(5,2) = sin(0.5) Using radians, sin(0.5) ≈ 0.4794 Final answer: PE_(5,2) ≈ 0.4794
Common mistakes to avoid
- Using degrees instead of radians for the sine function
- Forgetting that the exponent depends on the dimension index i, so each channel has a different wavelength
- Applying the formula to only one dimension and assuming it fully represents the whole positional encoding
Did You Know?
Sinusoidal encodings let Transformers generalize to sequence lengths never seen in training.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.