Skip to content
LearnMLByDoing
Free
07

Reinforcement Learning

Agents that learn from reward: Bellman equations, value iteration, Q-learning and the policy-gradient methods behind RLHF.

Lessons
3
Study time
1.7 h
Access
Pro track

Syllabus

  1. 7.1

    MDPs, Bellman equations and value iteration

    States, actions, rewards and discounting; watch value propagate backwards across a gridworld until the greedy policy stops changing.

    34 minProLive 3D lab and Python challengeintermediate
    • Formalise a problem as a Markov decision process
    • Derive the Bellman optimality equation and run value iteration
    • Extract a greedy policy and explain the role of the discount factor
  2. 7.2

    Q-learning, SARSA and exploration

    Learn action values from experience alone, balance exploration against exploitation, and see why off-policy learning walks the cliff edge.

    34 minProLive 3D lab and Python challengeintermediate
    • Implement tabular Q-learning with an epsilon-greedy policy
    • Contrast on-policy SARSA with off-policy Q-learning
    • Explain how DQN stabilises learning with replay and target networks
  3. 7.3

    Policy gradients, actor-critic and PPO

    Differentiate expected reward directly: REINFORCE, baselines and advantages, and the clipped objective that made PPO the default.

    36 minProLive 3D lab and Python challengeadvanced
    • Derive the policy-gradient theorem and the REINFORCE estimator
    • Reduce variance with baselines and advantage estimates
    • Explain PPO's clipped surrogate objective and its link to RLHF