Machine Learning &amp; AI

\hat{y} = \mathbf{w}^\top \mathbf{x} + b

Predicts a continuous value as a weighted sum of input features plus a bias.

Gradient Descent Update

\theta := \theta - \alpha\,\nabla_\theta J(\theta)

Iteratively moves parameters in the direction that most reduces the loss.

ReLU Activation

f(x) = \max(0, x)

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

f(x) = \max(\alpha x, x)

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

\tanh(x) = \dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

\sigma(z)_i = \dfrac{e^{z_i}}{\sum_{j} e^{z_j}}

Turns a vector of scores into a probability distribution that sums to 1.

Binary Cross-Entropy

L = -\big[y\log\hat{y} + (1-y)\log(1-\hat{y})\big]

Loss for binary classification comparing predicted probability to the true label.

Categorical Cross-Entropy

L = -\sum_{i} y_i \log \hat{y}_i

Multi-class loss summing the log-probability of the correct class.

Logistic Regression

\hat{y} = \dfrac{1}{1+e^{-(\mathbf{w}^\top \mathbf{x}+b)}}

Applies the sigmoid to a linear model to output a class probability.

Euclidean Distance (KNN)

d(\mathbf{p},\mathbf{q}) = \sqrt{\sum_i (p_i-q_i)^2}

Straight-line distance between two points; the default metric for k-nearest neighbours.

Cosine Similarity

\cos\theta = \dfrac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}

Measures the angle between two vectors; 1 means identical direction.

Entropy (Information)

H(S) = -\sum_i p_i \log_2 p_i

Measures impurity/uncertainty of a set; used to split decision trees.

Gini Impurity

G = 1 - \sum_i p_i^2

Probability of misclassifying a random sample; the default split metric in CART trees.

Information Gain

IG = H(S) - \sum_k \dfrac{|S_k|}{|S|}H(S_k)

Reduction in entropy achieved by splitting a node on a feature.

Naive Bayes Posterior

P(y\mid \mathbf{x}) \propto P(y)\prod_i P(x_i\mid y)

Classifies by assuming features are conditionally independent given the class.

L2 Regularization (Ridge)

J = \text{MSE} + \lambda\sum_j w_j^2

Adds a penalty on squared weights to reduce overfitting.

L1 Regularization (Lasso)

J = \text{MSE} + \lambda\sum_j |w_j|

Penalizes absolute weights, driving many to exactly zero for feature selection.

Hinge Loss (SVM)

L = \max(0,\,1 - y\,\hat{y})

Loss for support vector machines; zero once a point is correctly classified with margin.

SVM Margin

M = \dfrac{2}{\lVert \mathbf{w}\rVert}

Width of the separating margin an SVM maximizes between classes.

Perceptron Update

\mathbf{w} := \mathbf{w} + \eta\,(y-\hat{y})\,\mathbf{x}

The original learning rule that nudges weights when a prediction is wrong.

SGD with Momentum

v := \beta v + (1-\beta)\nabla J;\;\; \theta := \theta - \alpha v

Accelerates gradient descent by accumulating a velocity of past gradients.

Adam Optimizer

\theta := \theta - \alpha\,\dfrac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}

Adaptive optimizer combining momentum and per-parameter learning rates.

RMSProp

E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g_t^2

Scales the learning rate by a running average of recent squared gradients.

Exponential LR Decay

\alpha_t = \alpha_0\, e^{-k t}

Gradually lowers the learning rate as training progresses.

K-Means Objective (WCSS)

J = \sum_{k=1}^{K}\sum_{\mathbf{x}\in C_k}\lVert \mathbf{x}-\mu_k\rVert^2

Clustering minimizes the total squared distance of points to their cluster centre.

Batch Normalization

\hat{x} = \dfrac{x-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}

Normalizes layer inputs across a mini-batch to speed and stabilize training.

Dropout (Inverted)

\tilde{x} = \dfrac{x\odot m}{1-p}

Randomly zeroes activations during training to prevent co-adaptation.

TF-IDF

\text{tfidf} = tf \cdot \log\dfrac{N}{df}

Weights a word by its frequency in a document against its rarity across the corpus.

PCA (Covariance)

\mathbf{C} = \tfrac{1}{n}\mathbf{X}^\top\mathbf{X},\;\; \mathbf{C}\mathbf{v}=\lambda\mathbf{v}

Finds directions of maximum variance via eigenvectors of the covariance matrix.

Scaled Dot-Product Attention

\text{Attn}(Q,K,V)=\text{softmax}\!\left(\dfrac{QK^\top}{\sqrt{d_k}}\right)V

Lets each token weigh and aggregate information from all others — the heart of Transformers.

Positional Encoding

PE_{(pos,2i)}=\sin\!\left(\dfrac{pos}{10000^{2i/d}}\right)

Injects word-order information into a Transformer that otherwise ignores position.

Layer Normalization

\hat{x}=\dfrac{x-\mu}{\sqrt{\sigma^2+\epsilon}}\,\gamma+\beta

Normalizes across features within a single sample; standard in Transformers.

KL Divergence

D_{KL}(P\Vert Q)=\sum_i P(i)\log\dfrac{P(i)}{Q(i)}

Measures how one probability distribution differs from a reference distribution.

Bellman Equation

V(s)=\max_a\Big[R(s,a)+\gamma\sum_{s^{\prime}}P(s^{\prime}\mid s,a)V(s^{\prime})\Big]

Expresses the value of a state as the best immediate reward plus discounted future value.

Q-Learning Update

Q(s,a):=Q(s,a)+\alpha\big[r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a)\big]

Model-free reinforcement learning rule that learns action values from experience.

Discrete Convolution

(f*g)(t)=\sum_{\tau} f(\tau)\,g(t-\tau)

Slides a filter over an input to detect local patterns; the core of CNNs.