Machine Learning & AI

Core algorithms behind machine learning and AI — models, activation functions, loss functions, optimizers, deep learning and reinforcement learning.

Machine Learning & AI formulas capture the core mathematics behind how modern models learn, predict, and improve, from linear regression and gradient descent to activation functions, loss functions, and similarity measures. This category is useful for students revising theory, exam preparation, and anyone applying AI in real projects, because these formulas explain how algorithms make decisions, train efficiently, and measure performance.

Linear Regression Model

\hat{y} = \mathbf{w}^\top \mathbf{x} + b

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

\theta := \theta - \alpha\,\nabla_\theta J(\theta)

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

f(x) = \max(0, x)

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

f(x) = \max(\alpha x, x)

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

\tanh(x) = \dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

\sigma(z)_i = \dfrac{e^{z_i}}{\sum_{j} e^{z_j}}

Turns a vector of scores into a probability distribution that sums to 1.

Binary Cross-Entropy

L = -\big[y\log\hat{y} + (1-y)\log(1-\hat{y})\big]

Loss for binary classification comparing predicted probability to the true label.

Categorical Cross-Entropy

L = -\sum_{i} y_i \log \hat{y}_i

Multi-class loss summing the log-probability of the correct class.

Logistic Regression

\hat{y} = \dfrac{1}{1+e^{-(\mathbf{w}^\top \mathbf{x}+b)}}

Applies the sigmoid to a linear model to output a class probability.

Euclidean Distance (KNN)

d(\mathbf{p},\mathbf{q}) = \sqrt{\sum_i (p_i-q_i)^2}

Straight-line distance between two points; the default metric for k-nearest neighbours.

Cosine Similarity

\cos\theta = \dfrac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}

Measures the angle between two vectors; 1 means identical direction.

Entropy (Information)

H(S) = -\sum_i p_i \log_2 p_i

Measures impurity/uncertainty of a set; used to split decision trees.

Gini Impurity

G = 1 - \sum_i p_i^2

Probability of misclassifying a random sample; the default split metric in CART trees.

Information Gain

IG = H(S) - \sum_k \dfrac{|S_k|}{|S|}H(S_k)

Reduction in entropy achieved by splitting a node on a feature.

Naive Bayes Posterior

P(y\mid \mathbf{x}) \propto P(y)\prod_i P(x_i\mid y)

Classifies by assuming features are conditionally independent given the class.

L2 Regularization (Ridge)

J = \text{MSE} + \lambda\sum_j w_j^2

Adds a penalty on squared weights to reduce overfitting.

L1 Regularization (Lasso)

J = \text{MSE} + \lambda\sum_j |w_j|

Penalizes absolute weights, driving many to exactly zero for feature selection.

Hinge Loss (SVM)

L = \max(0,\,1 - y\,\hat{y})

Loss for support vector machines; zero once a point is correctly classified with margin.

SVM Margin

M = \dfrac{2}{\lVert \mathbf{w}\rVert}

Width of the separating margin an SVM maximizes between classes.

Perceptron Update

\mathbf{w} := \mathbf{w} + \eta\,(y-\hat{y})\,\mathbf{x}

The original learning rule that nudges weights when a prediction is wrong.

SGD with Momentum

v := \beta v + (1-\beta)\nabla J;\;\; \theta := \theta - \alpha v

Accelerates gradient descent by accumulating a velocity of past gradients.

Adam Optimizer

\theta := \theta - \alpha\,\dfrac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}

Adaptive optimizer combining momentum and per-parameter learning rates.

RMSProp

E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g_t^2

Scales the learning rate by a running average of recent squared gradients.

Exponential LR Decay

\alpha_t = \alpha_0\, e^{-k t}

Gradually lowers the learning rate as training progresses.

K-Means Objective (WCSS)

J = \sum_{k=1}^{K}\sum_{\mathbf{x}\in C_k}\lVert \mathbf{x}-\mu_k\rVert^2

Clustering minimizes the total squared distance of points to their cluster centre.

Batch Normalization

\hat{x} = \dfrac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}

Normalizes layer inputs across a mini-batch to speed and stabilize training.

Dropout (Inverted)

\tilde{x} = \dfrac{x\odot m}{1-p}

Randomly zeroes activations during training to prevent co-adaptation.

TF-IDF

\text{tfidf} = tf \cdot \log\dfrac{N}{df}

Weights a word by its frequency in a document against its rarity across the corpus.

PCA (Covariance)

\mathbf{C} = \tfrac{1}{n}\mathbf{X}^\top\mathbf{X},\;\; \mathbf{C}\mathbf{v}=\lambda\mathbf{v}

Finds directions of maximum variance via eigenvectors of the covariance matrix.

Scaled Dot-Product Attention

\text{Attn}(Q,K,V)=\text{softmax}\!\left(\dfrac{QK^\top}{\sqrt{d_k}}\right)V

Lets each token weigh and aggregate information from all others — the heart of Transformers.

Positional Encoding

PE_{(pos,2i)}=\sin\!\left(\dfrac{pos}{10000^{2i/d}}\right)

Injects word-order information into a Transformer that otherwise ignores position.

Layer Normalization

\hat{x}=\dfrac{x-\mu}{\sqrt{\sigma^2+\epsilon}}\,\gamma+\beta

Normalizes across features within a single sample; standard in Transformers.

KL Divergence

D_{KL}(P\Vert Q)=\sum_i P(i)\log\dfrac{P(i)}{Q(i)}

Measures how one probability distribution differs from a reference distribution.

Bellman Equation

V(s)=\max_a\Big[R(s,a)+\gamma\sum_{s^{\prime}}P(s^{\prime}\mid s,a)V(s^{\prime})\Big]

Expresses the value of a state as the best immediate reward plus discounted future value.

Q-Learning Update

Q(s,a):=Q(s,a)+\alpha\big[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a)\big]

Model-free reinforcement learning rule that learns action values from experience.

Discrete Convolution

(f*g)(t)=\sum_{\tau} f(\tau)\,g(t-\tau)

Slides a filter over an input to detect local patterns; the core of CNNs.