Q-Learning Update
Model-free reinforcement learning rule that learns action values from experience.
Formula
Variables
Example
The bracket is the temporal-difference error
How it works
This update rule adjusts the current estimate of the value of taking action a in state s by moving it a little toward a better estimate built from experience: the immediate reward plus the discounted value of the best action available in the next state. The difference inside the brackets is the prediction error, so if the outcome was better than expected the value goes up, and if it was worse it goes down. It works because repeated one-step updates gradually propagate information about future rewards back to earlier state-action pairs, even when the environment model is unknown.
When to use it
Use this when training an agent by trial and error in an environment where you can observe transitions and rewards but do not know the dynamics in advance. It is especially useful for discrete state-action problems such as grid worlds, simple game agents, and tabular control tasks.
Worked example, step by step
Suppose Q(s,a)=4.0, the learning rate \u03b1=0.2, the reward r=1, the discount \u03b3=0.9, and in the next state s' the best estimated action value is max_a' Q(s',a')=6.5. Compute the target: r + \u03b3 max_a' Q(s',a') = 1 + 0.9(6.5) = 1 + 5.85 = 6.85 Compute the error: 6.85 - 4.0 = 2.85 Scale by the learning rate: 0.2 \u00d7 2.85 = 0.57 Update the value: Q(s,a) := 4.0 + 0.57 = 4.57 Final answer: the new Q-value is 4.57
Common mistakes to avoid
- Using the value of the action actually taken in the next state instead of the maximum over next actions
- Forgetting to apply the discount factor to the next-state estimate
- Updating with the full error instead of multiplying by the learning rate
Did You Know?
Deep Q-Networks used this rule to learn Atari games directly from raw pixels in 2013.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.