Gradient Descent Update
Iteratively moves parameters in the direction that most reduces the loss.
Formula
Variables
Example
grad=0.5, alpha=0.1: step reduces theta by 0.05
How it works
This rule updates a model by nudging its parameters opposite to the gradient of the loss, because the gradient points in the direction of steepest increase. Multiplying by the learning rate controls how big each step is, so the model improves gradually instead of overshooting. Repeating this process can move the parameters toward a value that gives a smaller loss.
When to use it
Use this when training models whose performance is measured by a differentiable loss function, such as linear regression, neural networks, or logistic regression. It is the core update step in many optimization methods used to fit parameters from data.
Worked example, step by step
Given: θ = 2.40, α = 0.05, and ∇θJ(θ) = -0.80 Step 1: Substitute into the update rule θ := θ - α∇θJ(θ) Step 2: Compute the step size 0.05 × (-0.80) = -0.04 Step 3: Update the parameter θ := 2.40 - (-0.04) = 2.44 Final answer: the new parameter value is θ = 2.44
Common mistakes to avoid
- Adding the gradient instead of subtracting it
- Using a learning rate that is too large, which can cause the loss to increase or bounce around
- Forgetting that the gradient can be negative, so subtracting a negative number actually increases the parameter
Did You Know?
Almost every neural network on Earth is trained by some variant of gradient descent.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.
Binary Cross-Entropy
BasicLoss for binary classification comparing predicted probability to the true label.