How can a robot learn to manipulate physical objects simply by looking, listening, and practicing?

In this book, we answer that question by taking you through the complete engineering journey of building, training, and evaluating such a system. A robotic arm is mounted on a table holding several colored objects and two bins. It receives a single high-level command in plain English—such as "put the red cube in the right bin"—along with live feeds from two cameras and the current angles of its own joints.

The robot is never given object coordinates, 3D bounding boxes, or pre-programmed waypoints. From raw pixels and a sentence alone, it must interpret the scene, locate the requested object, coordinate its motors to reach and grasp it, and safely deliver it into the correct bin.

Everything in this project is developed inside a physics simulator called MuJoCo. Simulation provides a fast, safe environment where we can run thousands of trials before deploying onto physical hardware. But while the environment is simulated, the control system is entirely real: no human programmer wrote the rules that guide the arm. Instead, the robot learned to act through experience.

Before looking under the hood, let us start with the finished system in action. In Video 1.1, the robot receives a single instruction:

put the red cube in the right bin

Video 1.1 A complete 29-second attempt at real speed. Left: a fixed overhead camera showing the workspace. Right: a wrist camera looking down past the gripper.

The arm locates the red cube among the distractors on the table, descends, grasps it, carries it across the workspace, and drops it into the target bin.

How does a computer program turn camera pixels and words into continuous physical motion? Throughout this book, we explore both fundamental paths to solving this problem:

• Training from scratch: We build control policies from first principles using reinforcement learning, discovering firsthand what works, what breaks, and why learning physical manipulation purely by trial and error is so challenging.
• Using pretrained models: We take large vision-language-action (VLA) models—which already understand images and text—and fine-tune them with both supervised and reinforcement learning to master reliable physical control.

This book is the step-by-step account of how we built, tested, and trained both systems.

---

1.1 The Control Loop: Policy, Robot, and Environment

How does the robot accomplish this task in practice?

Beneath the surface, the system operates as a continuous closed loop executing ten times per second (10 Hz). As shown in Figure 1.1, the machine is organized into three connected parts: the top row contains the robotic agent—divided into the Policy (the brain) and the Robot (the body)—while the bottom row represents the Environment (the physical workspace).

Figure 1.1 The embodied control loop running at 10 Hz (100 ms per step). The Policy (top left) evaluates sensory observations and emits actions to the Robot (top right). The Robot executes physical motion on the Environment (bottom). Sensors continuously deliver the updated observation back to the Policy.

1. The Policy (Brain) decides: At each time step $t$, the policy $\pi$ evaluates the incoming observation $o_t$ (camera feeds, joint angles, and the user prompt $g$). Rather than planning a full trajectory in advance, it outputs an incremental action $a_t$: how far to shift the hand in 3D space $(\Delta x, \Delta y, \Delta z)$, how much to rotate the wrist $(\Delta \theta)$, and a target gripper opening.
2. The Robot (Body) moves: The UR5e arm and RG6 gripper receive this action command. Onboard joint motors drive toward the target, translating the policy's request into physical movement.
3. The Environment (World) updates: The physical workspace—containing the tabletop, the cubes, and the bins—reacts to the robot's motion. Contacts form between the fingers and the object, friction allows the fingers to lift the cube, and the objects shift in space.

As the physical state updates, the overhead and wrist cameras register the new scene, emitting the next observation $o_{t+1}$. This feedback loop runs ten times per second, continuously adjusting the robot's motion until the task is complete.

---

1.2 The Policy: Mapping Observations to Actions

In the control loop above, the policy ($\pi$) has a clear mathematical role: it is the function or model that maps what the robot perceives into what it should do next:

$$\text{Action} = \pi(\text{Observation})$$

Across the experiments in this book, this policy takes three very different forms:

• A hand-written expert: A deterministic rule-based controller programmed with geometric heuristics (we construct one in Chapter 3 to prove that the task is physically solvable).
• A policy trained from scratch: A compact neural network trained purely through reinforcement learning (such as SAC or PPO).
• A Vision-Language-Action (VLA) foundation model: A large multimodal neural network (such as the 3.5-billion-parameter model powering the robot in our opening demonstration) that natively processes vision and language tokens.

Regardless of internal architecture, every policy in our setup operates across the exact same interface:

What makes this manipulation problem demanding is what the policy is not given. It receives no 3D bounding boxes, no target coordinates, and no pre-computed grasp points. The policy must infer the physical structure of the workspace entirely from sensory feedback: recognizing that a cube has flat faces, figuring out that it must grasp securely before lifting, and delivering the object to the designated bin without collisions.

---

1.3 Emergent Recovery: Watching the Policy Fail and Retry

Looking closely at that demonstration reveals something unexpected: the robot did not succeed on its first attempt. The gripper closed and began to lift twice, but the cube remained on the table. Only on the third attempt was the grasp secure enough to complete the pick.

You can inspect this sequence step by step using the interactive player below:

Figure 1.2 Frame-by-frame breakdown of the 295-step attempt. Use the slider or buttons to inspect each grasp attempt alongside telemetry for jaw gap, hand height, and cube elevation.

Why did the first two attempts fail? In both cases, the arm began lifting before both gripper pads established symmetrical contact on the cube, causing it to slip out of the grasp.

The crucial engineering insight, however, is what happened next. A naive industrial automation script follows an open-loop plan: having failed the grasp, it would have carried empty air across to the bin, opened its empty jaws, and reported success.

Because our policy operates in a closed loop, it observes that the cube is still on the table. Without any hand-coded recovery logic—no if not grasped: retry() branch anywhere in the codebase—the policy autonomously descends, reorients, and attempts the grasp again until the cube is firmly held.

This self-correcting behavior is not programmed; it is an emergent property of training a reactive policy in a closed perception-action loop.

---

1.4 Why This Is Harder Than It Looks

Even when an attempt succeeds, training a system to reach this level of performance is notoriously difficult. Beneath the surface, autonomous manipulation presents four fundamental challenges in machine learning and robotics:

1. Perception without shortcuts:
   The policy never receives 3D bounding boxes or object coordinates. It observes only raw RGB pixel arrays. To successfully grasp a 45 mm cube, the policy must localize the object to within millimeters purely from images—interpreting depth, perspective, and lighting without hardcoded vision heuristics.

2. Delayed feedback and temporal credit assignment:
   In an episode spanning hundreds of sequential steps, the definitive signal of success arrives only at the end: does the cube land in the bin, or not? If a trial fails after 200 steps, which specific decision was responsible? Did the gripper misalign its approach at step 40, or did the fingers slip at step 180? Distributing credit across long continuous action trajectories is one of the central difficulties in reinforcement learning.

3. Generalization across scene variations:
   The workspace is never identical twice. In every trial, the target cube and distractors spawn at randomized positions and orientations, and the arm initializes in slightly different joint configurations. A policy cannot simply memorize a trajectory; it must learn generalized visual affordances that adapt to arbitrary scene layouts.

4. The Sim-to-Real gap:
   Developing policies in simulation provides safety, speed, and scale. However, simulated physics is inevitably an approximation of physical reality: friction models, contact dynamics, motor latencies, and camera optics all differ on real hardware. Training a policy in simulation that can successfully transfer to a physical UR5e arm without brittle failures requires careful modeling and robustness.

---

1.5 Roadmap of the Book

Rather than jumping straight to complex multi-billion-parameter models, this book follows the true sequence of engineering discoveries from the project. We begin with simple physical models and baseline tests, build up reinforcement learning from first principles, and only then integrate large foundation models.

The journey is structured into four main phases:

Phase I: Physical Foundations & Validation

• Chapter 2: The Robot and the Simulation — How the UR5e arm moves, how joint kinematics work, and why we simulate the cell in MuJoCo.
• Chapter 3: First, Show It Can Be Done — Constructing a hand-written geometric expert to prove the task is physically solvable and uncover subtle simulation bugs.

Phase II: Reinforcement Learning from Scratch

• Chapter 4: Learning from a Score — The reinforcement learning framework, value functions, policy gradients, and continuous action spaces.
• Chapter 5: Points for Standing Still — The perils of reward shaping: why our first learning agent chose to hover motionless above the table.
• Chapter 6: Imitation Learning — Teaching a policy to imitate expert demonstrations through supervised behavioral cloning.
• Chapter 7: Making RL Work — Modern off-policy reinforcement learning with SAC, critic stabilization, and solving the pick-and-place task from state.
• Chapter 8: Action Representations — How coordinate frames and rotation parameterizations impact learning speed.

Phase III: Vision & Foundation Models

• Chapter 9: The Perception Bottleneck — Why learning manipulation directly from raw pixels is so sample-inefficient.
• Chapter 10: Vision-Language-Action Models — The architecture of multimodal foundation models like Pi0 and how they ground language into physical actions.
• Chapter 11: Fine-Tuning with RL — Improving pretrained models through autonomous reinforcement learning in simulation.

Phase IV: Robustness & Practical Lessons

• Chapter 12: Domain Randomization & Visual Distortions — Preparing policies for the physical world through sensor noise, lighting variations, and dynamics jitter.
• Chapter 13: What We Learned — Practical lessons and engineering guidelines for building robot learning pipelines.

---

1.6 Prerequisites & How to Read This Book

This book is written for engineers, students, and researchers with a practical programming background:

• What you need: Comfortable working knowledge of Python, basic matrix operations (linear algebra), and elementary probability.
• What you do not need: Prior experience with robotics, control theory, or reinforcement learning. Every concept—from forward kinematics to policy gradient losses—is introduced and explained as it arises.

With the big picture established, we begin in Chapter 2 by looking closely at the machine itself: the UR5e robot, its gripper, and the physics of the simulation.