Machine Learning & AI

Advanced

Scaled Dot-Product Attention

Lets each token weigh and aggregate information from all others — the heart of Transformers.

Formula

\text{Attn}(Q,K,V)=\text{softmax}\!\left(\dfrac{QK^\top}{\sqrt{d_k}}\right)V

Variables

QQueries

KKeys

VValues

d_kKey dimension

Example

Scaling by sqrt(d_k) keeps gradients stable

Did You Know?

The 2017 paper "Attention Is All You Need" introduced this and launched the era of GPT and BERT.

transformer attention deep learning

Share this formula

More in Machine Learning & AI

Linear Regression Model

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Turns a vector of scores into a probability distribution that sums to 1.