Module 00: Introduction
In March 2016, a program called AlphaGo defeated Lee Sedol - one of the greatest Go players in history - four games to one.1 Go has more possible board positions than atoms in the observable universe. No amount of brute-force search could crack it. Instead, DeepMind used reinforcement learning: the program played millions of games against itself, learning from nothing but the rules and the outcome - win or lose.
The night before Game 4, Lee Sedol couldn't sleep. He later said he felt "helpless" - not because AlphaGo was faster, but because it played moves that no human had ever conceived in 3,000 years of Go history. In Game 2, Move 37, AlphaGo placed a stone that every expert in the room thought was a mistake. It turned out to be the most creative move of the match. The machine hadn't memorized human games - it had transcended them.
That moment changed everything. Not because RL was new (it wasn't - Sutton and Barto published the foundational textbook in 19982), but because it proved that an agent could learn superhuman strategy in a domain so complex that humans had spent millennia trying to master it.
RL is Everywhere (and You Didn't Notice)
Since AlphaGo, RL has quietly infiltrated industries you interact with daily:
-
ChatGPT uses RLHF (RL from Human Feedback) to turn a raw language model into something that actually follows instructions and doesn't immediately go off the rails. Without RL, GPT-4 would be an impressive autocomplete that happily writes bioweapon instructions in the same helpful tone it uses for cookie recipes.3
-
Your Netflix recommendations are partially driven by RL. The system treats each recommendation as an action, your watch/skip as the reward, and optimizes for long-term engagement - not just "will they click?" but "will they still be subscribed next month?"4
-
DeepMind's Waymo uses RL to train autonomous driving policies in simulation before deploying them on real roads. The agent experiences millions of years' worth of driving in simulation, including rare events that a human test driver might encounter once in a lifetime - like a shopping cart rolling into traffic at 3 AM.5
-
Robots at Berkeley learn to pick up objects they've never seen before, purely from trial and error in simulation, then transfer those skills to the real world. The robot doesn't know what a coffee mug is. It just knows that certain grip patterns lead to "object lifted" rewards.6
-
Google's data centers cut cooling energy consumption by 40% using an RL agent that learned to manage airflow better than any human operator.7 The system controls over 120 variables simultaneously - something no human can do intuitively.
-
Quantitative hedge funds like Two Sigma and Renaissance Technologies experiment with RL for portfolio optimization - teaching agents to trade in environments where the "rules" shift daily and yesterday's winning strategy can be tomorrow's catastrophe.8
-
Drug discovery at companies like Recursion Pharmaceuticals uses RL to navigate the astronomical space of possible molecular modifications, learning which chemical tweaks are most likely to improve a drug's efficacy without wrecking its safety profile.9
The common thread? An agent interacts with an environment, receives rewards, and gets better over time. That's the entire field in one sentence. The rest of this course is the details - and the details are wild.
In supervised learning, you have a dataset with correct answers. A cat-or-dog classifier knows the right answer for every training image. In RL, you have no answers - just a world, actions, and consequences. The agent must discover good behavior through experience. It's closer to how children learn: not from labeled examples, but from trying things and seeing what happens. A child doesn't need a dataset of 10,000 labeled "hot stove" images. They touch it once.
A Brief History of Learning from Consequences
RL didn't appear out of thin air. The intellectual lineage runs deep:
-
1950s: Arthur Samuel at IBM built a checkers program that learned by playing against itself. He coined the term "machine learning" to describe it. IBM's management was nervous - they didn't want people to think computers could "think."10
-
1989: Gerald Tesauro created TD-Gammon, a backgammon agent trained with temporal-difference learning (which you'll learn in Module 04). It reached world-class play and discovered strategies that human experts initially dismissed as mistakes - until they tried them and found they actually worked.11
-
1992: Sutton published his PhD thesis on temporal-difference learning, laying the theoretical groundwork for modern RL.2
-
2013: Mnih et al. at DeepMind published the DQN paper - a single neural network that learned to play 49 different Atari games from raw pixels. The same architecture, the same hyperparameters, the same learning algorithm. Pong, Breakout, Space Invaders - it crushed them all. The agent had never seen a video game before.12
-
2016: AlphaGo defeats Lee Sedol. The world pays attention.1
-
2019: OpenAI Five defeats the world champions in Dota 2, a game with partial observability, long time horizons, and teamwork among five agents.13
-
2022: ChatGPT launches, built on RLHF. Suddenly everyone's grandmother has an opinion on reinforcement learning.3
-
2024: RL agents manage real-world plasma in nuclear fusion reactors, keeping the superheated matter stable for longer than any hand-tuned controller.14
Notice the pattern: every few years, RL does something that experts said was decades away. That's what makes this field addictive to work in.
What You'll Build in This Course
By the end of these 16 modules, you'll be able to:
- Implement Q-learning, SARSA, DQN, policy gradients, PPO, and SAC from scratch
- Train agents that balance poles, land rockets, and navigate mazes
- Understand the math deeply enough to read (and reproduce) research papers
- Debug the infamously unstable training of deep RL agents - you'll know why your reward curve looks like a seismograph during an earthquake and what to do about it
- Apply RL to real problems - from game AI to LLM alignment to robotics
But first, we need to set up your lab.
Learning Objectives
- Install Python + core RL libraries.
- Understand the Gymnasium (OpenAI Gym successor) environment interface.
- Run your first RL agent (random policy) on CartPole.
- Learn how to log and plot episode rewards.
1. Prerequisites
Make sure you have:
- Python 3.12+
- pip or uv (package manager)
- A virtual environment (venv, conda, or uv)
Always use a virtual environment. Your future self will thank you when two projects need different PyTorch versions. This is not a suggestion - it's a survival strategy.
Create a Virtual Environment
# Option A: uv (recommended - it's fast)
uv venv .venv
source .venv/bin/activate
# Option B: classic venv
python -m venv rl_env
source rl_env/bin/activate # macOS/Linux
# rl_env\Scripts\activate # Windows
2. Install Required Libraries
pip install gymnasium numpy matplotlib pygame torch
| Library | Purpose |
|---|---|
gymnasium | RL environments (replaces old gym) |
numpy | Numerical computing (arrays, math) |
matplotlib | Plotting rewards, values, etc. |
pygame | Required for rendering (e.g., visualizing CartPole) |
torch | PyTorch (for deep RL later) |
Confirm it works:
python -c "import gymnasium as gym; print(gym.__version__)"
▶Expected Output
1.1.1
3. The RL Loop - One Slide, Infinite Depth
Every RL algorithm, from the simplest to the most sophisticated, follows the same loop:
1. Observe the state
2. Choose an action
3. Execute the action in the environment
4. Receive a reward and a new state
5. Learn something from this experience
6. Repeat
That's it. AlphaGo does this. ChatGPT's RLHF training does this. The random agent you're about to run does this (it just skips step 5, which is why it's terrible).
Sutton and Barto call this the agent-environment interface,2 and it's one of the most productive abstractions in all of computer science. The environment can be a board game, a physics simulator, a hospital, a stock market, or a conversation with a human. The agent doesn't care - it just sees states, takes actions, and collects rewards.
Here's what's beautiful about this: when you learn to think in terms of "state, action, reward," you start seeing RL problems everywhere:
| Situation | State | Action | Reward |
|---|---|---|---|
| Playing chess | Board position | Move a piece | +1 win, -1 loss |
| Driving a car | Speed, lane, traffic | Accelerate, brake, steer | +1 alive, -1000 crash |
| Studying for an exam | Knowledge level, time left | Study topic A or B, or sleep | Grade on exam |
| Managing a restaurant | Tables occupied, kitchen queue | Seat guests, assign waitstaff | Revenue + tips |
| Training a dog | Dog's current behavior | Give treat, say command | Dog sits (finally) |
The beauty of the RL framework is that the same algorithms work across all of these. An algorithm doesn't know if it's playing Go or managing a supply chain - it just sees numbers and optimizes.
4. Why RL is Hard (A Preview of Suffering)
Before you get too excited, let me be honest about what you're getting into. RL is the most exciting area of ML, but it's also the most frustrating. Here's a taste of what makes it hard:
The credit assignment problem. Your agent makes 1,000 moves in a chess game and loses. Which move was the mistake? Move 37? Move 421? All of them? RL algorithms have to figure out which actions in a long sequence actually mattered. In supervised learning, the loss tells you exactly what went wrong for each example. In RL, you just get a single number at the end: "you lost."
Exploration vs. exploitation. Your agent found a strategy that scores 50 points. Should it keep doing that, or try something risky that might score 100... or 0? This is the fundamental dilemma of RL, and it has no perfect solution. It's the same dilemma you face when choosing a restaurant: go to your reliable favorite, or try the new place with the weird name? You'll dive deep into this in Module 02 with multi-armed bandits.
Sample inefficiency. Modern deep RL agents can need millions of environment interactions to learn things a human picks up in minutes. DQN needed about 200 million Atari frames - roughly 38 days of real-time gameplay - to reach human performance.12 You can't afford that if your environment is the real world (you don't get to crash 200 million cars to learn to drive).
Instability. Training curves in RL look nothing like the smooth descent of supervised learning. They look like this: up, up, up, plateau, sudden collapse, recovery, new plateau, collapse again, finally up again. You'll learn to not panic when this happens. Henderson et al.15 showed that the same RL algorithm with different random seeds can produce wildly different results - a finding that shook the field and changed how we report results.
Reward hacking. Give an agent a poorly designed reward and watch it find "creative" solutions. A famous example: a boat-racing agent that discovered it could score more points by spinning in circles and collecting bonus items than by actually finishing the race.16 Another: a robot trained to walk that discovered it could move faster by growing very tall and then falling forward repeatedly. Technically not walking, but the reward function didn't specify that.
This is all ahead of you. For now, let's run some code.
5. Test Your Installation
import gymnasium as gym
import numpy as np
import time
# Create a simple environment
env = gym.make("CartPole-v1", render_mode="human")
# Reset the environment
state, info = env.reset()
print("Initial state:", state)
print("Action space:", env.action_space)
print("Observation space:", env.observation_space)
# Run a random agent for 100 steps
for step in range(100):
action = env.action_space.sample()
state, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
print(f"Episode ended at step {step}")
state, info = env.reset()
time.sleep(0.02)
env.close()
▶Expected Output
Initial state: [ 0.0273 0.0195 -0.0479 -0.0301]
Action space: Discrete(2)
Observation space: Box([-4.8 -inf -0.42 -inf], [4.8 inf 0.42 inf], (4,), float32)
Episode ended at step 14
Episode ended at step 31
Episode ended at step 48
Episode ended at step 63
Episode ended at step 89
A pygame window opens showing the cart and pole. The pole wobbles randomly and falls over every 10-25 steps.
You should see a window with a pole on a cart, wobbling around like it had one too many. That's your first RL environment. The pole falls over almost immediately because our agent is choosing random actions - it has zero understanding of physics, balance, or consequences. Over the next 15 modules, we'll fix that.
What you're watching is genuinely the same setup as AlphaGo, just with a much simpler problem. The cart sees its position, velocity, pole angle, and angular velocity (4 numbers). It can push left or right (2 actions). It gets +1 reward for every timestep the pole stays upright. When the pole falls past 12 degrees or the cart rolls off the screen, the episode ends. Simple rules, and yet: try to write a hand-coded controller for this. It's doable, but surprisingly fiddly. RL agents find solutions without any physics knowledge - they just discover that "when the pole tilts right, push right" through trial and error.
6. Understanding Gymnasium
Gymnasium17 is the standard interface for RL environments. It was created by the Farama Foundation as the maintained successor to OpenAI Gym. Every environment follows the same API:
env = gym.make("CartPole-v1")
obs, info = env.reset() # start a new episode
for _ in range(1000):
action = env.action_space.sample() # pick an action
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()
env.close()
Key components
| Component | What it does |
|---|---|
env.reset() | Returns initial observation and info dict |
env.step(action) | Returns (obs, reward, terminated, truncated, info) |
env.action_space | Defines valid actions |
env.observation_space | Defines the shape/range of observations |
The Environments Zoo
Gymnasium ships with dozens of environments. Here are the ones we'll use most:
| Environment | State | Action | The Challenge |
|---|---|---|---|
CartPole-v1 | Continuous (4D) | Discrete (2) | Balance a pole on a cart. The "Hello World" of RL. |
MountainCar-v0 | Continuous (2D) | Discrete (3) | A car stuck in a valley. It can't drive straight up the hill - it must learn to swing back and forth to build momentum. This one will teach you about sparse rewards (you get -1 every step and 0 only when you reach the flag). Most beginners' first encounter with "my agent isn't learning anything." |
FrozenLake-v1 | Discrete (16) | Discrete (4) | Navigate a frozen lake without falling in holes. With is_slippery=True, the ice is treacherous - you only go where you intended 1/3 of the time. Perfect for learning about stochastic environments. |
LunarLander-v2 | Continuous (8D) | Discrete (4) | Land a spaceship on a landing pad. Fuel costs reward, crashing costs a lot. A satisfying visual environment that's harder than it looks. |
Pendulum-v1 | Continuous (3D) | Continuous (1D) | Swing a pendulum upright and keep it there. Your first continuous-action problem - you can't just pick from {left, right}, you have to choose how much torque. |
As the course progresses, we'll graduate to more complex environments: Atari games from raw pixels, MuJoCo robotics tasks where you control simulated robot joints, and eventually multi-agent environments where your agents have to cooperate or compete.
7. Your First Agent (Random Policy)
Let's write a proper RandomAgent class - even though it's terrible at its job:
import gymnasium as gym
import numpy as np
class RandomAgent:
def __init__(self, action_space: gym.spaces.Space) -> None:
self.action_space = action_space
def act(self, obs: np.ndarray) -> int:
return self.action_space.sample()
def run_random_agent(env_name: str = "CartPole-v1", episodes: int = 5) -> list[float]:
env = gym.make(env_name)
agent = RandomAgent(env.action_space)
rewards = []
for ep in range(episodes):
obs, info = env.reset()
total_reward = 0.0
done = False
while not done:
action = agent.act(obs)
obs, reward, terminated, truncated, info = env.step(action)
total_reward += reward
done = terminated or truncated
rewards.append(total_reward)
print(f"Episode {ep + 1}: reward = {total_reward}")
env.close()
print(f"Average reward: {np.mean(rewards):.2f}")
return rewards
if __name__ == "__main__":
run_random_agent()
▶Expected Output
Episode 1: reward = 22.0
Episode 2: reward = 14.0
Episode 3: reward = 37.0
Episode 4: reward = 19.0
Episode 5: reward = 11.0
Average reward: 20.60
Over 500 episodes, the distribution looks like this - heavily right-skewed, clustered around 15-25:
A random agent on CartPole is like a toddler trying to balance a broomstick - entertaining but not effective.
Why is this useful? Because it gives us a baseline. In RL, you always want to know: "Is my fancy algorithm actually better than random?" You'd be surprised how often the answer is "barely." Henderson et al.15 demonstrated that many published RL results barely beat well-tuned baselines, and that random seeds alone can account for massive performance differences. Always run your random baseline first. It keeps you honest.
8. Plotting Episode Rewards
The reward curve is your most important diagnostic tool in RL. Get comfortable reading these.
import matplotlib.pyplot as plt
rewards = run_random_agent("CartPole-v1", episodes=200)
plt.figure(figsize=(10, 4))
plt.plot(rewards, alpha=0.4, label="Raw")
# Rolling average to see the trend
window = 20
rolling = [np.mean(rewards[max(0,i-window):i+1]) for i in range(len(rewards))]
plt.plot(rolling, linewidth=2, label=f"Rolling avg ({window} eps)")
plt.xlabel("Episode")
plt.ylabel("Total Reward")
plt.title("Random Agent on CartPole")
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
You'll see noisy rewards, almost always below 50, with a flat rolling average. Our goal by the end of this course? 500 consistently (the CartPole-v1 max). That flat line of noise you see now will eventually become a beautiful upward curve - the unmistakable signature of an agent learning.
The rolling average trick is something you'll use constantly. Raw RL rewards are noisy because of the stochastic nature of both the environment and the policy. A single episode can be lucky or unlucky. The rolling average tells you the real story.
Here's what a random agent's reward curve actually looks like - hover over points to see individual episode scores:
9. Capturing Frames (Headless Mode)
On a server without a display, use rgb_array to capture frames as numpy arrays:
import gymnasium as gym
import matplotlib.pyplot as plt
env = gym.make("CartPole-v1", render_mode="rgb_array")
env.reset()
env.step(env.action_space.sample())
frame = env.render() # returns an RGB numpy array
plt.imshow(frame)
plt.title("CartPole Frame")
plt.axis("off")
plt.show()
env.close()
This is how you'll record training videos and log visual results to TensorBoard or W&B later. There's nothing quite like watching your agent go from random flailing to smooth, deliberate control - it's the RL equivalent of watching a baby learn to walk, compressed into a few minutes of video.
10. The Landscape of RL Algorithms
Before we dive into the modules, here's a map of the territory. Don't worry about understanding all of this now - we'll cover every one of these in detail. But it helps to see where you're going:
RL Algorithms
├── Model-Free
│ ├── Value-Based (learn V or Q, derive policy)
│ │ ├── Monte Carlo ← Module 04
│ │ ├── SARSA / Q-Learning ← Module 04
│ │ └── DQN ← Module 05
│ ├── Policy-Based (learn policy directly)
│ │ ├── REINFORCE ← Module 06
│ │ └── PPO, SAC, TD3 ← Module 07
│ └── Actor-Critic (learn both)
│ └── A2C, A3C, PPO, SAC ← Modules 06-07
└── Model-Based (learn a model of the environment)
├── Dyna ← Module 10
└── MuZero, World Models ← Module 10
The two big questions that divide RL algorithms are: (1) Do you learn a value function, a policy, or both? (2) Do you try to model how the environment works, or just learn from raw experience?
Here's an interactive view - each bubble is an algorithm you'll learn, positioned by how sample-efficient it is vs. how well it performs. Click on any point for details:
David Silver's RL lecture series at UCL18 is the best video companion for this course. If you're the kind of person who learns from lectures, start watching alongside Module 01.
11. What's Coming Next
Now that your lab is set up and your first (terrible) agent is running, here's a preview of the journey:
- Module 01: The math that makes RL precise - MDPs, Bellman equations, and why discount factors exist. We'll solve small problems exactly before learning to approximate.
- Module 02: Multi-armed bandits - the purest form of the exploration-exploitation dilemma. Should you go to your favorite restaurant, or try the new place with the weird name?
- Modules 03-04: Dynamic programming, Monte Carlo, and TD learning - the classical methods that Sutton and Barto call "the core of RL."2
- Modules 05-07: Deep RL - where neural networks meet the agent-environment loop, and things get interesting (and occasionally catch fire).
- Module 08: The module your future self will thank you for - reward design, debugging, and all the things that go wrong in practice.
- Module 11: RLHF - how the same RL ideas that play Go now align ChatGPT.
By Module 07, you'll have a PPO agent that consistently scores 500 on CartPole. By Module 14, you'll be reproducing results from real research papers.
12. Common Issues & Fixes
| Problem | Solution |
|---|---|
ModuleNotFoundError: No module named 'gymnasium' | pip install gymnasium |
No module named 'pygame' | pip install pygame |
render_mode not working | pip install --upgrade gymnasium |
| Black screen / no window | Use render_mode="human" and check your display server |
env.P not found on FrozenLake | Only available with is_slippery=False or via env.unwrapped.P |
| WSL/SSH: No display available | Use render_mode="rgb_array" and save frames as images |
13. Pro Tips from the Trenches
- Use
gymnasium, not the oldgympackage. The old package has unfixed bugs and is unmaintained. Gymnasium is the official successor. - Always check
terminatedandtruncatedseparately.terminatedmeans the episode ended naturally (fell off a cliff).truncatedmeans it hit a time limit (survived but out of time). Conflating these is a common source of bugs that can silently ruin your training. - Start with
CartPole-v1orFrozenLake-v1. They're fast (1000x faster than Atari), simple enough to debug, and rich enough to illustrate every core concept. - Print everything when debugging. Print
state,action,reward,doneon every step. RL bugs are sneaky - your agent might be "learning" to exploit a reward bug rather than solving the task. - Version control your experiments. You'll run hundreds of experiments. Future-you needs to know which code produced which results. Git commit before every experiment run.
Milestone Checklist
- Installed libraries and confirmed Gymnasium works
- Ran a random agent on CartPole
- Logged and plotted rewards over multiple episodes (with rolling average)
- Understood the step/reset API
- Can explain the agent-environment interface in your own words
Exercises
- FrozenLake showdown: Run the random agent on
FrozenLake-v1withis_slippery=Truefor 1000 episodes. What's the win rate? Now tryis_slippery=False. Why is the difference so dramatic?
▶Solution
# Slippery version
rewards = run_random_agent("FrozenLake-v1", episodes=1000)
wins = sum(1 for r in rewards if r > 0)
print(f"Slippery win rate: {wins/1000:.1%}")
Slippery win rate: 1.4%
Non-slippery win rate: 6.2%
The slippery version is dramatically harder because the agent only goes where it intended 1/3 of the time. Even if you stumble upon a decent path, the ice sends you sliding into holes. The non-slippery version is still terrible (random walks rarely find the goal in a 4x4 grid), but at least the agent's actions are deterministic.
Here's how random agents compare across all the environments in this exercise:
- LunarLander: Try
LunarLander-v2(you may needpip install gymnasium[box2d]). What's the average random reward? Watch the rendering - the random agent's landing attempts are genuinely hilarious.
▶Solution
rewards = run_random_agent("LunarLander-v2", episodes=100)
Average reward: -178.42
The random agent scores around -150 to -250 per episode. It fires thrusters randomly, wastes fuel, crashes into the ground at high speed, and occasionally flies off screen entirely. The negative reward comes from fuel usage (-0.3 per frame for main engine) and crashing (-100). Landing on the pad gives +100 to +140, but a random agent essentially never manages it.
- The "always right" agent: Modify
RandomAgentto always pick action 1 (push right) on CartPole. Does it do better or worse than random? What does this tell you about the environment?
▶Solution
class AlwaysRightAgent:
def act(self, obs):
return 1 # always push right
# Run it the same way as RandomAgent
Average reward: 9.40
Worse than random! The always-right agent scores ~9-10, while random scores ~20. This makes sense: always pushing right sends the cart off the right edge quickly, and the pole tilts left with no correction. A random agent at least accidentally corrects sometimes. This tells you that CartPole requires responsive actions - you need to react to the pole's angle, not blindly repeat one action.
- Environment designer: Pick a real-world problem you care about (managing a coffee shop, training for a marathon, studying for exams). Define its RL components: What's the state? What are the actions? What's the reward? What makes it harder than CartPole? Write a paragraph about it. (This is not a throwaway exercise - thinking in RL terms is the most important skill you'll develop.)
▶Example Solution: Coffee Shop
State: Time of day, day of week, current inventory levels (beans, milk, pastries), number of customers waiting, number of staff on shift, weather outside, nearby events.
Actions: Adjust staffing (call someone in / send someone home), place supply orders, change menu prices, offer a promotion.
Reward: Revenue minus costs (staff wages + supply costs + waste from expired items). Penalty for long customer wait times (they leave and don't come back).
Why it's harder than CartPole: (1) The state space is enormous and partially observable (you don't know how many customers will arrive). (2) Actions have delayed effects (ordering supplies today affects tomorrow). (3) The environment is non-stationary (seasonal trends, new competitors). (4) Multiple competing objectives (profit vs. customer satisfaction vs. staff happiness).
- History dive: Read the first chapter of Sutton & Barto (free online).2 They tell the story of how RL grew from three separate threads: trial-and-error learning from animal psychology, optimal control from engineering, and temporal-difference learning from computer science. Write down one thing that surprised you.
References
Footnotes
Was this page helpful?