Search Results

8 results for “deep learning”

Formulae (8)

f(x) = \max(0, x)

Rectified Linear Unit: outputs the input if positive, else zero.

f(x) = \max(\alpha x, x)

A ReLU variant that lets a small gradient flow for negative inputs.

\tanh(x) = \dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}

Squashes input to the range (-1, 1); zero-centred activation.

\hat{x} = \dfrac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}

Normalizes layer inputs across a mini-batch to speed and stabilize training.

\tilde{x} = \dfrac{x\odot m}{1-p}

Randomly zeroes activations during training to prevent co-adaptation.

\text{Attn}(Q,K,V)=\text{softmax}\!\left(\dfrac{QK^\top}{\sqrt{d_k}}\right)V

Lets each token weigh and aggregate information from all others — the heart of Transformers.

PE_{(pos,2i)}=\sin\!\left(\dfrac{pos}{10000^{2i/d}}\right)

Injects word-order information into a Transformer that otherwise ignores position.

\hat{x}=\dfrac{x-\mu}{\sqrt{\sigma^2+\epsilon}}\,\gamma+\beta

Normalizes across features within a single sample; standard in Transformers.