Learn
A4.3.6 | REINFORCEMENT LEARNING (HL ONLY)
01 | LEARN FROM THE CONSEQUENCES OF A CHOICE
Imagine a simulated delivery robot at a junction. One action finishes the task immediately for a small reward. Another enters a corridor, initially costing a point, but makes a larger later reward possible. Which behaviour should it learn?
Reinforcement learning is learning to make decisions through interaction with an environment. An agent chooses actions, observes consequences and receives reward signals. Experience is used to improve its decision-making policy toward greater expected cumulative reward.
The learning signal is not a supplied correct action for every situation. A reward can arrive later, after several choices. This makes reinforcement learning different from simply following a programmed route or classifying labelled examples.
02 | THE AGENT IS THE DECISION-MAKER
The agent is the system choosing actions. For a robot, it can be the control software that decides where to move. For a game, it can be the program choosing moves.
The agent receives information about its situation, chooses an available action and may update what it has learned after seeing the result. It need not resemble a person or have human intentions.
The designer still defines the task, available actions, observations and reward scheme. The agent learns within those choices; it does not automatically discover what humans really intended.
03 | THE ENVIRONMENT RESPONDS
The environment is the system with which the agent interacts. It determines what happens after an action and supplies the next observation and reward. In a simulator, code implements those consequences.
A robot environment can include the layout, obstacles and movement dynamics. The controller is the agent; the world and relevant simulation mechanics form the environment.
An action does not always have the same outcome in real tasks. Wheels may slip and other participants may move. Our small demonstration is deterministic so you can follow each update exactly.
04 | STATE AND OBSERVATION: WHAT SITUATION ARE WE IN?
A state describes the relevant situation. In the toy environment, it is simply Start, Corridor or Goal. Those names describe positions in the task, not category labels the agent is trying to predict.
An observation is the information supplied to the agent. A real robot may observe camera images, sensor readings and battery level without knowing the full physical state of the world.
For clarity, the activity gives the agent its complete toy state. In more complex tasks, hidden information or missing history can make decisions harder. A useful state representation must capture what matters for the decision.
05 | ACTIONS ARE THE AVAILABLE CHOICES
An action is a choice the agent can take. At Start, our robot can select Direct or Corridor. In Corridor, it can select Finish or Return. Goal ends the episode and has no further action.
These are symbolic choices in a simulation. A real controller might choose motor speeds, steering angles or a higher-level movement command. Action spaces can be discrete or continuous.
The agent must choose among valid actions for the current state. Learning does not mean that every imaginable action is available or physically possible.
06 | REWARD IS FEEDBACK, NOT A COMPLETE INSTRUCTION
A reward is a numerical signal returned after an action. It can be positive, zero or negative. The design encourages some outcomes relative to others.
A reward of −1 for entering a corridor does not mean the agent must never enter it. A later outcome might compensate for that cost. Conversely, a positive immediate reward does not automatically make a choice best overall.
Reward is not the same as a labelled instruction saying which action was correct. It provides feedback about an outcome under the defined scoring system. The agent must learn which sequences tend to produce good returns.
07 | THE AGENT–ENVIRONMENT LOOP
| Stage | What happens in the toy task |
|---|---|
| Observe | Agent receives current state: Start |
| Choose | Its current policy chooses Corridor |
| Act | Environment moves it to Corridor |
| Receive feedback | Environment returns reward −1 and next state Corridor |
| Learn | Agent updates its estimate for that state–action experience |
| Repeat | Choose another action until the episode ends |
The loop connects a choice to its consequences. Merely drawing movement or adding up a player’s score is not enough to demonstrate learning: some stored decision rule or estimate must change through experience.
The interactive activity below updates a genuine small action-value table. You can inspect those numbers rather than being asked to assume that a moving marker is learning.
08 | A POLICY CONNECTS SITUATIONS TO ACTIONS
A policy describes how the agent chooses actions from the information available. It can select one action deterministically or assign probabilities to several actions.
An initial policy might always select Direct at Start because the available estimates are tied and its tie rule favours the first action. After suitable experience, its preferred action may change to Corridor.
The policy is not necessarily a complete prewritten route. It can make a fresh decision at each state. A value-based learner can derive a policy by favouring actions with the best current estimates.
09 | REWARD NOW AND RETURN OVER TIME
The objective is usually to maximise expected cumulative reward, often called return, rather than greedily taking the largest immediate reward. Expected means averaging over possible outcomes when the environment or policy is uncertain.
For a finite episode with no discounting, add all its rewards. A sequence −1, −1, +8 has cumulative reward 6. A single immediate +2 has return 2.
An agent may therefore learn to accept a short-term cost to reach a better later outcome. That requires learning the consequences of sequences, not just memorising which immediate reward is largest.
10 | READ THE TWO-ROUTE ENVIRONMENT
| Current state | Action | Next state | Immediate reward | Ends the task? |
|---|---|---|---|---|
| Start | Direct | Goal | +2 | Yes |
| Start | Corridor | Corridor | −1 | No |
| Corridor | Finish | Goal | +8 | Yes |
| Corridor | Return | Start | −1 | No |
The Direct route gives +2 in one action. Corridor followed by Finish gives −1 + 8 = 7 over two actions. Repeated Return actions add costs and delay completion.
This is a deliberately tiny learning problem, not a physical robot controller. The transition table lets you understand the task; the demonstration’s learner updates its estimates from experienced transitions rather than planning with this table.
11 | EPISODES PROVIDE REPEATED EXPERIENCE
An episode is one run from a starting situation until a task-ending condition. Here it starts at Start and ends on reaching Goal. Another episode resets the position while retaining the learned table.
If resetting the position also erased the estimates every time, the agent would not accumulate knowledge across attempts. Resetting the entire learner is a separate operation in the activity.
A practical training run also needs a step limit to avoid endless loops. The activity cuts an episode off after 20 actions if Goal has not been reached. It distinguishes that artificial cutoff from reaching Goal.
12 | DELAYED REWARD CREATES A CREDIT-ASSIGNMENT PROBLEM
The action Corridor gives −1 now, while Finish later gives +8. The agent must learn that the earlier choice can enable the later reward. This is a simple form of the credit-assignment problem.
A learner that compares only immediate rewards would prefer Direct’s +2 over Corridor’s −1. A learner considering future return can favour Corridor after gaining enough relevant experience.
Different reinforcement-learning methods propagate or estimate future value in different ways. The demonstration uses one small tabular Q-learning example; it is not the definition of all reinforcement learning.
13 | EXPLORATION: TRY TO LEARN SOMETHING NEW
Exploration means choosing actions to gather information, including actions not currently believed to be best. It gives the agent a chance to discover outcomes that its present estimates miss.
If the agent always chooses Direct after finding its +2, it may never discover Finish’s larger reward. Initial ignorance can become a permanent limitation when the policy never tests an alternative.
Exploration has costs: it can take longer or produce worse outcomes during training. In real systems, exploration must be constrained appropriately; a safe simulator is a useful place to learn before considering physical deployment.
14 | EXPLOITATION: USE WHAT HAS BEEN LEARNED
Exploitation means selecting an action believed to have the best expected return according to the current estimates. It uses experience to make a preferred choice.
Early estimates can be wrong. Exploiting them too soon can lock the agent into a modest route. Exploring all the time, however, means frequently ignoring a better route even after learning it.
The exploration–exploitation trade-off is the balance between gaining information and benefiting from what is already known. The right balance depends on the environment, uncertainty and whether the goal is training or deployment.
15 | AN EPSILON-GREEDY POLICY MAKES THE BALANCE VISIBLE
One simple strategy is epsilon-greedy. With probability ε, choose a random valid action; otherwise choose a currently highest-valued action. Epsilon is a hyperparameter controlling the exploration branch.
With ε = 0.20, 20% of decisions use the random-action branch on average over many decisions. The random branch may still choose the same action as the greedy branch, so this does not mean exactly 20% of actions differ from the preferred action.
The activity uses a reproducible pseudorandom stream. It reports whether each decision used the random or greedy branch. Exact ties in the greedy branch select Direct at Start and Finish in Corridor. All initial value estimates are zero.
16 | WHAT THE AGENT STORES IN THIS EXAMPLE
A Q-value is an estimate of future return for taking an action in a state and then continuing according to the target behaviour represented by the method. The demonstration stores four estimates: two actions at Start and two in Corridor.
After seeing (state, action, reward, next state), Q-learning moves the corresponding estimate toward a target made from immediate reward plus estimated future value. Reaching Goal gives no further future term.
The robot position, total reward and learned estimates are different things. A single episode can score poorly while still provide useful information. This is why you should inspect the stored values and evaluate a policy separately from its exploratory training behaviour.
17 | OPTIONAL DETAIL: ONE UPDATE YOU CAN CHECK
The demonstration uses learning rate α = 0.5 and discount factor γ = 0.9. Its update is new Q = old Q + α × (target − old Q), where target is reward + γ × the largest next-state Q-value, or just reward on reaching Goal.
If the untrained agent takes Direct, old Q is 0 and the terminal reward is +2. Target = 2, therefore new Q = 0 + 0.5 × (2 − 0) = 1. Another identical experience moves it from 1 to 1.5.
The learning rate determines the size of this estimate adjustment. The discount factor weights later reward. Understanding the agent–environment loop and cumulative reward is the syllabus priority; this equation makes the activity’s actual learning inspectable rather than being a required derivation.
18 | DISCOUNTED RETURN IS NOT THE SAME AS THE RAW SCORE
With γ = 0.9, Corridor then Finish has discounted return −1 + 0.9 × 8 = 6.2 when measured from Start. Its ordinary undiscounted episode reward is 7. Both exceed Direct’s return of 2 in this task.
The Q-table estimates discounted future return under this method, while the episode score adds raw rewards. Do not expect a Start Q-value to converge to 7 when the method discounts the later reward.
A larger discount factor gives future outcomes more weight relative to the present. Delayed rewards, uncertain outcomes and long action sequences can all make learning more difficult; discounting does not mean that future rewards are guaranteed.
19 | WATCH THE AGENT LEARN
Begin with no exploration, take several steps or train a batch, and inspect which action it keeps choosing. Then reset all learning, enable exploration and train again. The batch button completes 100 further episodes using actual Q-value updates.
Changing ε affects subsequent training choices and retains the current table. “Reset all learning” clears estimates and restarts the same pseudorandom stream.
The activity is genuine tabular Q-learning in a fixed toy environment, not a neural network or a physical robot. It does not guarantee learning from every short run. The displayed update explains the most recent transition; batch summaries describe the completed batch.
Goal is terminal, so its future value is zero. At an artificial 20-action cutoff, the update still uses the next-state estimate before resetting; the cutoff is not treated as a genuine terminal state with no future value.
20 | COMPARE TRAINING WITH EVALUATION
Training behaviour includes exploratory actions, so its episode scores can vary even when the agent has useful estimates. To inspect its learned preference, the evaluation button follows the current greedy policy from Start without updating the table or consuming training randomness.
After exploration has revealed the corridor route, a greedy evaluation may select Corridor then Finish, returning 7 raw reward and 6.2 discounted reward. With no exploration from an empty table, the specified tie rule can keep the learner on Direct for reward 2.
This is an explanation of the toy result, not evidence of general navigation ability. Real evaluation should test relevant starting states, changed conditions and multiple runs; one successful episode in a fixed simulator is not a reliability guarantee.
21 | REWARD DESIGN CAN PRODUCE UNINTENDED BEHAVIOUR
Suppose a designer gives a positive reward for every metre travelled but forgets to reward successful delivery appropriately. An agent might learn to move repeatedly rather than complete the task. That exploits the scoring system while missing the human goal.
Rewards need to reflect the intended outcome and constraints. Sparse reward, arriving only at a distant goal, can make useful discovery difficult. Adding intermediate rewards can help learning but can also change the behaviour being optimised.
The agent optimises the signal and environment it experiences, not an unstated moral or business objective. Test for loopholes, relevant failures and changes in conditions rather than assuming a large training return means the task has been solved properly.
22 | APPLICATIONS AND LIMITS
| Application | Agent choices | Possible reward signal | Limit to investigate |
|---|---|---|---|
| Game playing | Moves in a game state | Winning and task-related outcomes | Training game and opponents may not represent new play |
| Robot control | Motion or control actions | Task completion with costs for time or energy | Physical exploration can cause damage; simulator transfer is imperfect |
| Resource scheduling | Which task receives resources next | Completed work, delay and resource-use trade-offs | A proxy score may omit fairness or important constraints |
These are possible reinforcement-learning formulations, not claims that every game, robot or scheduler must use RL. A fixed route program or a feedback controller can act without learning from rewards.
RL can require substantial experience and computation. Hidden state, changing environments, poor reward design and limited safe exploration can all reduce usefulness. Choose a method because the sequential decision problem warrants it, not because a system contains a robot.
23 | CHECK THE LEARNING CONCEPT
Match each statement to its main concept. Distinguish the world, the choice mechanism and the feedback used to improve it.
24 | DESCRIBE THE WHOLE INTERACTION
A complete description connects the terms: the agent observes its situation, chooses an action according to a policy, receives a reward and next observation from the environment, and uses experience to improve future decisions. The goal is greater expected cumulative reward across sequences, not merely the highest immediate score.
Use the route example: “Corridor initially costs a point but enables a later reward of eight. Exploration can reveal that sequence, while exploitation uses the learned preference. Always taking the early +2 can prevent discovery of the better route.”
Include a limit: “The policy can only learn from the experience and reward design available. A high return in one fixed toy environment does not establish safe performance in a different real world.”
Further reading: agent–environment interactions and an inspectable Q-learning update.