Scaled Dot-Product Attention
Lets each token weigh and aggregate information from all others — the heart of Transformers.
Formula
Variables
Example
Scaling by sqrt(d_k) keeps gradients stable
How it works
Scaled dot-product attention computes how relevant each key is to a given query by taking their dot product, then turning those scores into probabilities with softmax. Dividing by sqrt(d_k) prevents the raw dot products from growing too large as the key/query dimension increases, which would make the softmax overly peaky and learning unstable. The resulting weights are then used to form a weighted average of the value vectors, so each output token becomes a content-dependent mixture of information from the whole sequence.
When to use it
Use this when you need a model to choose, for each position in a sequence, which other positions matter most, such as in language modeling, machine translation, vision transformers, or any sequence-to-sequence task. It is especially useful when long-range dependencies matter and simple fixed-window methods are too limiting.
Worked example, step by step
Take one query and three key-value pairs: Q = [1, 2], K1 = [1, 0], K2 = [0, 1], K3 = [1, 1] V1 = [10, 0], V2 = [0, 10], V3 = [5, 5], and d_k = 2 Step 1: Compute raw scores Q·K^T: Q·K1 = 1·1 + 2·0 = 1 Q·K2 = 1·0 + 2·1 = 2 Q·K3 = 1·1 + 2·1 = 3 Step 2: Scale by sqrt(d_k) = sqrt(2) ≈ 1.414: [1/1.414, 2/1.414, 3/1.414] ≈ [0.707, 1.414, 2.121] Step 3: Apply softmax: e^0.707 ≈ 2.028, e^1.414 ≈ 4.113, e^2.121 ≈ 8.341 Sum = 2.028 + 4.113 + 8.341 = 14.482 Weights ≈ [2.028/14.482, 4.113/14.482, 8.341/14.482] ≈ [0.140, 0.284, 0.576] Step 4: Weighted sum of values: Attn(Q,K,V) = 0.140[10,0] + 0.284[0,10] + 0.576[5,5] = [1.40, 0] + [0, 2.84] + [2.88, 2.88] = [4.28, 5.72] Final answer: Attn(Q,K,V) ≈ [4.28, 5.72]
Common mistakes to avoid
- forgetting to divide by sqrt(d_k), which makes attention scores too large for high-dimensional vectors
- applying softmax to the values V instead of the similarity scores QK^T
- mixing up the roles of queries, keys, and values: keys determine matching, values are what get averaged
Did You Know?
The 2017 paper "Attention Is All You Need" introduced this and launched the era of GPT and BERT.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.