Bellman Equation
Expresses the value of a state as the best immediate reward plus discounted future value.
Formula
Variables
Example
gamma near 1 values the long-term future
How it works
The Bellman equation states that the value of a state is determined by the best action you can take now plus the discounted value of what is likely to happen next. It works because of the principle of optimality: if a plan is optimal overall, then whatever remains after the first decision must also be optimal from the next state onward. The transition probabilities average over all possible next states, so the formula captures uncertainty rather than assuming a single outcome.
When to use it
Use this when analyzing sequential decision problems where outcomes depend on both current choices and uncertain future states, such as reinforcement learning, Markov decision processes, robotics, or game AI. It is especially useful when you want to compute an optimal policy or evaluate how valuable a state is under long-term consequences.
Worked example, step by step
Suppose an agent is in state s and can choose between two actions. Action a1: immediate reward R(s,a1)=5, transitions to s'1 with probability 0.7 and value V(s'1)=10, or s'2 with probability 0.3 and value V(s'2)=4. Action a2: immediate reward R(s,a2)=6, transitions to s'3 with probability 1.0 and value V(s'3)=7. Let gamma = 0.9. For a1: Expected future value = 0.7(10) + 0.3(4) = 7 + 1.2 = 8.2 Discounted future value = 0.9(8.2) = 7.38 Total value = 5 + 7.38 = 12.38 For a2: Expected future value = 1.0(7) = 7 Discounted future value = 0.9(7) = 6.3 Total value = 6 + 6.3 = 12.3 Take the maximum: V(s) = max(12.38, 12.3) = 12.38 So the Bellman equation says the optimal value of state s is 12.38, achieved by choosing a1.
Common mistakes to avoid
- Forgetting to multiply future value by the discount factor gamma
- Using only one next state instead of the probability-weighted sum over all possible next states
- Taking the maximum over next states instead of over actions
Did You Know?
Richard Bellman coined "dynamic programming" — and the "curse of dimensionality" — in the 1950s.
Share this formula
More in Machine Learning & AI
View allLinear Regression Model
BasicPredicts a continuous value as a weighted sum of input features plus a bias.
Gradient Descent Update
BasicIteratively moves parameters in the direction that most reduces the loss.
ReLU Activation
BasicRectified Linear Unit: outputs the input if positive, else zero.
Leaky ReLU
BasicA ReLU variant that lets a small gradient flow for negative inputs.
Tanh Activation
BasicSquashes input to the range (-1, 1); zero-centred activation.
Softmax
BasicTurns a vector of scores into a probability distribution that sums to 1.