Gradient Descent Update
Iterative optimization step to minimize a cost function.
Formula
Variables
Example
Step size is controlled by learning rate \alpha
How it works
This update rule moves the parameter vector in the direction that most rapidly reduces the cost, which is the negative of the gradient. The gradient points uphill, so subtracting a small multiple of it nudges the parameters downhill on the loss surface. The learning rate controls how far you move each step: too large can overshoot the minimum, while too small makes convergence slow.
When to use it
Use this whenever you are training a differentiable model by iteratively minimizing a loss function, such as linear regression, logistic regression, neural networks, or matrix factorization. It is especially useful when the objective is too large or complex for a closed-form solution.
Worked example, step by step
Suppose a model has current parameters θ = [2.0, -1.0] and the gradient of the cost at that point is ∇J(θ) = [3.0, -4.0]. Let the learning rate be α = 0.1. Compute the step: α∇J(θ) = 0.1 × [3.0, -4.0] = [0.3, -0.4]. Update each parameter: θ_new = θ - α∇J(θ) = [2.0, -1.0] - [0.3, -0.4]. So θ_new = [1.7, -0.6]. The parameters have moved opposite the gradient, which should lower the cost if α is chosen appropriately.
Common mistakes to avoid
- Adding the gradient instead of subtracting it, which performs ascent rather than descent
- Using a learning rate that is too large and causes oscillation or divergence
- Forgetting that θ and ∇J(θ) must have matching dimensions so the subtraction is valid
Did You Know?
Choosing the learning rate is one of the most important tuning decisions.
Share this formula
More in Data Science & ML
View allMean Squared Error (MSE)
BasicAverage of squared differences between predicted and actual values.
Root Mean Squared Error (RMSE)
BasicSquare root of MSE, in the same units as the target variable.
Mean Absolute Error (MAE)
BasicAverage of absolute differences between predictions and actual values.
Coefficient of Determination (R²)
IntermediateProportion of variance in the target explained by the model.
Classification Accuracy
BasicFraction of predictions that are correct.
Precision
IntermediateOf all positive predictions, the fraction that were correct.