Machine Learning & AI
Basic

Gradient Descent Update

Iteratively moves parameters in the direction that most reduces the loss.

Formula

θ:=θ−α ∇θJ(θ)\theta := \theta - \alpha\,\nabla_\theta J(\theta)

Variables

\thetaParameters
\alphaLearning rate
\nabla_\theta JGradient of loss

Example

grad=0.5, alpha=0.1: step reduces theta by 0.05

Did You Know?

Almost every neural network on Earth is trained by some variant of gradient descent.

optimizationtraininggradient

Share this formula

More in Machine Learning & AI

View all

Linear Regression Model

Basic

Predicts a continuous value as a weighted sum of input features plus a bias.

ReLU Activation

Basic

Rectified Linear Unit: outputs the input if positive, else zero.

Leaky ReLU

Basic

A ReLU variant that lets a small gradient flow for negative inputs.

Tanh Activation

Basic

Squashes input to the range (-1, 1); zero-centred activation.

Softmax

Basic

Turns a vector of scores into a probability distribution that sums to 1.

Binary Cross-Entropy

Basic

Loss for binary classification comparing predicted probability to the true label.