Learning to Navigate, Failing to Recover

The task is called ObjectNav. Put a robot somewhere in a house it has never seen. Tell it what to look for: a bed, a sofa, a fridge. It has to walk there using only what its cameras show, and say “done” when it is close enough. No map. No GPS. No coordinates of the target. Just pictures and a sentence.

People find this easy. A fridge is in the kitchen, a bed is in a bedroom, and a door usually leads somewhere. A learned policy has to pick all of that up from data.

I spent two months on this. The post is long because I want it to be useful to someone who tries the same thing. I explain each method in a few sentences when it first appears, then say what it did. Every number comes from a logged run, and where a number turned out to be untrustworthy I say so, because that happened more than once.

01The task

The houses come from ProcTHOR-10K, a set of 10,000 procedurally generated homes that run in the AI2-THOR simulator. They come pre-split: 10,000 for training, 1,000 for validation, 1,000 for test. Every house has real rooms, doors, furniture and clutter. The robot is a Stretch, a wheeled robot with two RGB cameras: one looking forward, one looking down at the floor in front of it.

Each step, the robot picks one of seven actions:

An episode has 250 steps. Eight target categories: sofa, bed, television, fridge, toilet, chair, houseplant, countertop. The episode is a success if the robot says end while an instance of the target is visible and within two metres. That rule will matter a lot later, because for the first two weeks it was a different rule, and the different rule was wrong.

Two numbers are used throughout. Success rate is the share of episodes that end correctly. SPL is success weighted by path length: a success along the shortest possible route scores 1, a success after a long detour scores less. I mostly quote success, because SPL moved with it every time.

02The plan: copy an expert, then improve on it

Three stages, each one a test of the one before.

1. A privileged expert. A planner that cheats. It can read the simulator's navigation mesh, so it knows the shortest walkable path from anywhere to the target. It records what it sees and what it does. The policy will never have this information. It only gets the recordings.
2. Imitation learning. Train a network to predict the expert's action from the expert's camera images. This is behaviour cloning: plain supervised learning, where the label at each step is “what the expert did here”.
3. Reinforcement learning. Take that network, let it act in the simulator on its own, reward it for getting closer to the target and for stopping in the right place, and update it towards what earned reward. The hope is that RL fixes what imitation cannot: recovering from the policy's own mistakes, which the expert never makes and so never demonstrates.

The network is SPOC's EarlyFusionCnnTransformer. It is worth knowing its shape, because it explains several results below.

It has 227 million parameters. 203 million are frozen: the SigLIP image encoders and the text encoder are pretrained and never updated. Only 23 million, about 10%, are trained. The frozen part decides what the robot sees. The trainable part decides what it does with that. Keep that split in mind. When I later tried to fix perception with RL, the reason it could not work is right here.

03The expert, the data, and the first policy

The expert is SPOC's shortest-path planner, adapted to ProcTHOR houses. It picks a reachable instance of the target, computes the navmesh path, converts it to discrete moves and turns, re-plans if a move fails, and turns to face the target before saying end. Only successful episodes are saved.

Running it over the training houses gave the first dataset, V5: 29,055 episodes across 9,685 houses. Every episode stores both camera videos, the action at each step, the robot's position, and the target.

Behaviour cloning, and one adjustment

Behaviour cloning is a classifier. Input: the last 50 steps of images, goal text and previous actions. Output: a probability for each of the seven actions. Loss: cross-entropy against the expert's action. That is all.

One problem showed up immediately. About 67% of the expert's actions are m, move ahead. A classifier that always says m is 67% accurate and useless. So I weighted the loss by the inverse square root of each action's frequency. Rare actions like end count more. This is standard, and later I found it did not matter much either way.

The first policy scored well on the numbers you can compute offline:

Then I put it in the simulator and let it drive.

This is the first lesson, and it is the oldest one in imitation learning. Predicting the expert's label on the expert's own frames is easy. Driving is not. Every small wrong action puts the robot somewhere the expert never went, so the next frame is one the policy has never seen, and the errors compound. Offline F1 and closed-loop success are different measurements. From here on I only trusted the second.

A side quest: does the training code even work?

Before changing anything, I wanted to know the pipeline could learn at all. The standard check is an overfit test: give it 4, 16 or 32 examples, train for a long time, and see if it memorises them. It should reach perfect accuracy.

Four examples: memorised. Sixteen and thirty-two: no. That made no sense for a 227-million-parameter network. The cause was hidden augmentation. SPOC's data loader randomly shifted the time index of every sequence on every read, so the “same” 32 examples had different positional embeddings every epoch. Turning that off for the overfit test gave macro F1 of 1.0 on 32 samples. The trainer was fine. An hour of checking saved me from blaming the model for a loader.

04The score was wrong

I watched the failures frame by frame. In many of them the robot was standing in front of a perfectly good toilet, sofa, or television, and was being scored as a failure. In one house with five televisions, it stopped in front of one, and the scorer was measuring its distance to a different one, 12 metres away.

The success rule I had inherited was: within 1.5 m of the one annotated instance, facing it within 6 degrees. That is not ObjectNav. Finding any television should count. I changed the rule to what SPOC uses: any instance of the category is visible in the forward camera within 2 m. Visibility is answered by the simulator itself, which handles occlusion.

Nothing about the policy changed. It went from 0.200 to 0.433 because the question changed. Every conclusion I had drawn before this point was thrown away, because it was drawn from a number that did not measure the task.

From here I also fixed a benchmark: 147 validation episodes in 49 houses the policy never trained in, always the same episodes, so runs could be compared. On that benchmark the first policy scored 0.592.

05Five things I tried on imitation

With a benchmark I could trust more, I looked at what the 0.592 policy got wrong. 147 episodes, 60 failures, in three kinds: it said end in the wrong place (32), it got wedged against something and never recovered (23), or it ran out of steps (5). Watching the wrong stops, one pattern stood out: dark rectangles. Laptops, dark desks, framed pictures, all called “television”.

Idea 1: drop the previous action

The policy gets its own last action as an input. I suspected a copycat loop: it repeats what it just did because that is what it just did. So during training I randomly replaced the previous-action input with a blank token, 20% of the time. The policy would have to look at the images instead.

It did exactly what it was meant to. Spinning episodes went from 3 to 1. Opening rotations got shorter. And success fell from 0.592 to 0.531. The copycat dependency was load-bearing. “Keep doing what you were doing” was standing in for a “keep going” signal the policy had no other way to get.

Idea 2: unfreeze part of the image encoder

The television confusion is a perception error, and perception lives in the frozen encoder. So I unfroze the last four SigLIP blocks, 28 million parameters, at a tenth of the learning rate, and trained on the full dataset.

Success fell to 0.381. Television failures got worse, 16 to 29. With 29,000 episodes there is not enough signal to re-specialise a pretrained vision encoder; there is just enough to damage it. This ruled out touching the encoder for the rest of the project, and it is why the 203 million frozen parameters stayed frozen.

Idea 3: the robot had never seen anyone reverse

Twenty-three of the failures were the robot wedged in a door frame, pushing forward until the simulator gave up. The only way out of a door frame is backwards. I checked the data. The expert plans on a navmesh, so it always has a forward path, so it never emits b. Zero times in 29,055 episodes. The policy's F1 for b was exactly 0.000 in every run, and it produced b zero times in 5,485 closed-loop actions.

You cannot learn an action from data that does not contain it. So I added recovery episodes to the expert: about 3,300 episodes that start with the robot deliberately jammed against geometry, so the expert has to reverse before it can navigate. That became dataset V8. The share of b in the data went from 0% to 0.93%, and closed-loop the policy started using it in about 30 of 147 episodes. Wedged failures fell from 29 to 22. Better, not fixed. I will come back to why.

Idea 4: where the expert stops. The most expensive lesson.

After I changed the success rule, the expert was still stopping by the old rule. So 6% of the demonstrations ended in a place the new scorer calls a failure. The data was teaching the policy to stop in the wrong spot.

The obvious fix: make the expert stop the instant the new rule becomes true. I regenerated the whole dataset that way and trained on it. Success: 0.177. The worst result of the project.

Here is what happened. If the expert stops the moment the criterion flips, every demonstration of end sits right on the boundary of the success region. The policy learns that boundary. But its perception is not perfect, so at test time it lands a few centimetres off, and a few centimetres off the boundary is outside. Wrong stops went from 32 to 88 out of 147.

The rule that worked is stricter than the scorer: the target has to be roughly centred in the frame and large enough in the image. So the expert stops a little inside the region, with margin. I compared strict against loose at two small data sizes before regenerating everything: strict won both times, 0.156 vs 0.082 and 0.272 vs 0.184.

Idea 5: train longer, and stop trusting the checkpoint picker

The strict-rule dataset, V7, trained with the old settings did not beat 0.592. All three runs had early-stopped after two to four epochs. So I raised the epoch limit to 25 and the early-stopping patience to effectively never. Success went to 0.633. On V8, with the recovery episodes, it went to 0.687.

And a trap on the way. The trainer saves best.ckpt, the checkpoint with the best offline macro F1, and last.ckpt. Macro F1 averages over 20 action classes, 13 of which never occur in this task, so most of it is noise. On the winning run it picked a checkpoint that scored 0.490 over the one that scored 0.687.

The 0.687 policy is the end of the imitation story. Its 46 remaining failures: 22 wedged, 19 wrong stops, 5 out of steps. A privileged expert on a separate hard benchmark scored 0.983, so the episodes were solvable. The policy was the limit.

06Reinforcement learning, first attempt

Now the idea was to let the policy improve itself. Here is PPO in three sentences. The policy acts in the simulator and collects a few thousand steps of experience. For each step, compute an advantage: was this action better or worse than the policy expected? Then nudge the policy to make better-than-expected actions more likely, but only a little, so it does not jump off a cliff.

Two additions to the imitation network. A value head: a small extra output that predicts how much reward is coming, which is the “expected” in the advantage. And a KL limit: a cap on how much the action probabilities may change per update, measured as the KL divergence between old and new.

The reward:

Four runs from the 0.592 imitation policy, 20,000 steps each, with different learning rates and an entropy bonus. The best reached 0.612, and won 10 episodes against the imitation policy's 7. That is not a difference. Higher learning rates collapsed into 200-step episodes that never stopped. The entropy bonus made episodes longer and worse. Training longer made everything decay.

The agent optimises the rules as written

One thing the RL runs did learn: spinning. Rotating costs nothing under this reward. No collision, no wrong stop, and standing still is nearly free because progress only pays for reaching a new closest distance. An unsure policy could spin forever and lose almost nothing.

This is the first of several times the agent found a hole in my rules. It is not being clever. It is doing exactly what the reward says. Every hole below was found by watching videos, not by reading curves.

07Rebuilding the RL task

Rather than keep tuning, I stopped and asked what could be wrong. Five things were.

Is the optimiser right?

First the cheapest check. I ran the same PPO code on a five-cell corridor: start at 0, reach 4, +1 on arrival, -0.05 per step. A tiny MLP. It solved it within two updates. The optimiser was fine. I also ran Stable-Baselines3's PPO, an independent implementation, on my environment as a control. It behaved the same way mine did. Every later failure was looked for in the task, not the algorithm.

A real bug in the KL check

The KL limit was being checked on every minibatch of four samples, and one minibatch over the line aborted the whole update. An estimate from four samples is mostly noise. Measured: the average KL was ten times below the limit, the noisiest minibatch sat right at it, and 47% of updates were being thrown away. The fix checks the average once per epoch.

Distance through walls

The progress reward used straight-line distance to the target. Straight lines go through walls. For a bed in the next room, the reward is largest at the wall nearest the bed, so that is where the agent went, and pressed.

Geodesic just means: the length of the shortest path a robot can actually walk. The simulator can compute it. With that one change, a task that had zero successes in 858 attempts had 39 in 532.

Reversing through the house

The rule that ends an episode when the robot is stuck counted only failed forward moves. Reversing into a wall was free. A from-scratch policy discovered this and drove backwards through the entire house: 30% of its actions were b, against 8% m. It still reached the target.

Success the agent cannot find

With end required, a fresh policy has to say end in the right place by chance before it can learn anything about stopping. Two degenerate policies appeared: stop immediately every episode, or never stop and collect progress reward forever. I added a task variant, no-stop, where arriving within two metres ends the episode automatically. It is an easier task and not the benchmark, but it let me ask whether RL could improve navigation at all, separately from stopping.

08What RL could and could not do

From scratch: far too slow

A CNN trained by PPO from random weights reached 0.85 on one fixed bed episode after 120,000 steps. On random starts in one house it reached about 0.15 to 0.23 and stayed there. Fine-tuning the imitation policy reached 0.85 on the same fixed episode in 6,600 steps. Twenty times fewer. Starting from imitation was not optional.

Fine-tuning on the no-stop task: it works

Forgetting, and the anchor

The first multi-target run got better overall and lost the fridge completely: 4 of 5 to 0 of 9. A rare skill traded for common ones. This is catastrophic forgetting. The fix is an anchor: keep a frozen copy of the original imitation policy, and add to the loss the KL divergence between the current policy and that copy. The per-update KL limit stops big jumps within one update; the anchor stops slow drift across a hundred updates. With the anchor, the gains landed where imitation was weakest: fridge 1 to 5, television 7 to 11, houseplant 5 to 9.

Two limits

It did not transfer. On 80 episodes in five houses neither policy had seen: imitation 0.650, fine-tuned 0.675. PPO won 10, imitation won 8. A tie. Training in one house makes a specialist for that house. The useful part is the negative half: it did not get worse elsewhere, so the anchor preserved the original skill.

It was the wrong task. Every number above is no-stop. The real benchmark requires end. When I put end back, the value head was the problem: it starts random, nearly all reward is one rare +10 at the end, so early advantages are noise, and noise damages a policy that is already good. I added critic warmup: freeze the actor for the first six updates and train only the value head. It reduced the decay. It did not remove it.

Then I ran paired tests on the real task, and got this sequence:

Tie, +0.10, tie, +0.065. I reported the +0.10 as a result before test 3 ran. Test 3 was the same checkpoint on twice as many fresh episodes and it found nothing. That sequence is what a small or absent effect looks like when you measure it repeatedly on samples of 100 to 200. I could not tell if the effect was real, and the reason I could not tell was about to become clear.

09The benchmark did not repeat

The 0.687 imitation policy, re-run with identical settings on the identical 147 episodes, scored 0.639. Then 0.565. Fourteen runs of one unchanged checkpoint spanned 0.558 to 0.687.

The cause: Python's global random number generator was never seeded, and the evaluation depended on it. The obvious consumer was the goal sentence. SPOC picks a verb at random, so “find a bed” and “locate a bed” are different inputs to a text-conditioned policy. The wording differed in 129 of 147 episodes between two runs. But forcing each verb across all episodes gave 0.565 to 0.578, a spread of only 0.014. The wording was a symptom.

The RNG state itself was the cause. With the wording fixed, changing only the seed gave 0.565, 0.646, 0.646. 33 of 147 episodes flip depending on the seed. Twenty-two percent of the benchmark was a coin toss. I never fully traced where in the forward pass the RNG is consumed. I seeded it per episode, which makes a run repeatable, and moved on. Repeatable is not the same as correct. It freezes one draw from a distribution 0.08 wide.

A measurement that can be trusted

Three rules from here on, and every number below follows them.

1. Three seeds, report the mean and the range. Never one run.
2. Paired episodes. Both policies run the same episodes, in one process, under the same random state. The score of either alone is noisy. The difference on the same episode is much less so.
3. A paired test. Ignore the episodes where both succeed or both fail. Count the ones that disagree: A won and B lost, or the reverse. If the effect is real, the count is lopsided. McNemar's exact test gives the probability of a split that lopsided by chance. Because the three seeds repeat the same 147 episodes, I cluster by episode so the same episode is not counted three times.

And a control. Run a policy against itself under this protocol. The wins should split evenly. They did: 11-12, over 441 trials, with 6.6% of trials disagreeing with themselves. That 6.6% is the noise floor. Any effect has to be read against it.

10What the failures actually were

Weeks of stop-task RL rested on an assumption: a wrong stop means the policy stopped at the wrong moment. I had never checked. So I replayed all 19 wrong-stop episodes of the 0.687 policy, feeding the recorded actions back into the simulator and logging at every step whether the target was visible.

Not one was a mistimed stop. In all 19, the policy announced arrival somewhere the target had never once been in view. Ten of them stopped more than three metres out. So 80% of the failures were navigation: wedged, or never found the room. The stop-reward experiments had been aimed at a problem that did not exist.

The stop-task question, answered properly

With the paired protocol, the question I had spent six weeks on took seven GPU-hours. Imitation against imitation-plus-PPO, one house, 450 paired episodes, three seeds: +0.022, 29 wins against 19, p = 0.19. Not significant. PPO in one house with this setup does not reliably improve the stop-required task. Two seeds lean positive, one negative, which is what a small or absent effect looks like.

11A rule beat RL

All 110 actions that tripped the “stuck” rule on the benchmark were m. Not one turn, not one reverse. And there is a simple reason. The policy has no collision input. When a move is blocked, both camera frames are unchanged, and the previous action is still m. From the policy's point of view nothing happened, so it does the same thing. Five repeats, and the episode ends. Zero of 101 successful episodes contained a single collision; 28 of 46 failures did.

So, a rule. Applied at test time only, nothing trained: if a move was blocked, forbid that same action on the next step. The policy must pick its second choice. The ban clears as soon as any action succeeds, so it never touches normal driving. I call it the collision reflex.

+0.031, 37 wins against 9, p = 0.00004. The first result in the project that replicated on fresh episodes.

Why a rule reached significance when PPO, at the same size, did not

PPO changed many episodes, about half better and half worse. The reflex changed few episodes, and nearly all for the better. A paired test reads the imbalance, not the count. A narrow intervention with a mechanism you understand, that predicts which episodes will change, is easier to prove than a broad one of equal average size. That is the most useful methodological lesson of the whole project.

RL on top of the reflex: an exact null

With the reflex on, wedging disappears and the dominant failure becomes not finding the target. RL had improved search on the no-stop task. So the hypothesis was: PPO failed before because it trained on episodes that ended at a wall after a median 47 steps; give it the reflex during training and it learns navigation instead. I ran it. Both arms with the reflex. -0.002, 27 wins against 28, p = 1.0. The hypothesis was specific, cheap, and wrong.

12Making PPO work: four changes

Change 1: detach the value head

The value head was reading the transformer's output directly, and its loss gradient flowed back into the transformer. Even during critic warmup, when the actor was “frozen”, the critic's regression errors were rewriting the representations the actor depends on. That is what had been collapsing runs.

With the trunk protected, one-house PPO on the stop task gave +0.022 with 16 wins against 6, p = 0.05. Suggestive, and for the first time the policy never fell below imitation on any seed.

Change 2, tried and dropped: a coverage reward

A bonus of 0.05 for every new half-metre grid cell the robot enters, to encourage exploring rooms. It won more episodes (18) but also lost more (10), because the robot sometimes explored instead of finishing. +0.018, p = 0.18. Pure progress reward with the detached trunk was tighter. Dropped.

Change 3: a stop gate

A second test-time rule. The policy is not allowed to say end unless it has assigned that action at least 50% probability, and at least five steps have passed. It removes zero-effort instant stops. Stacked on the detached PPO policy with the reflex, on one house: +0.064 against imitation, 38 wins to 9. Measured alone, later, the gate does nothing detectable: +0.005, p = 0.57. I kept it and do not credit it.

Change 4: train across houses

Training in one house makes a specialist. So I rotated training across 25 houses, a fresh random start and target every episode, 60,000 steps. Evaluated on the 147 episodes in 49 houses it had never seen, both arms carrying reflex and gate so RL is the only difference:

This is the RL result of the project. Reinforcement learning, on top of imitation, on the real stop-required task, on houses it never trained in, adds about seven points. The head-to-head of 25 houses against 1 house is +0.034 at p = 0.0501, one ten-thousandth from the line, so “more houses help” is suggestive and not proven.

13More houses looked useless. It was the budget.

If 25 houses beat 1, do 100 beat 25? I trained 100-house and 200-house policies with everything else the same, same 60,000 steps, and compared head to head.

Flat. 100 vs 25: +0.016, p = 0.45. 200 vs 100: -0.007, p = 0.86. The obvious conclusion was that house diversity stops paying at 25.

The obvious conclusion was wrong, and I am glad I checked before writing it down. Every training episode is freshly sampled, so with a fixed 60,000 steps, more houses means fewer episodes per house: 45 for the 25-house run, 6 for the 200-house run. I had not varied the number of houses. I had varied the number of visits per house, and called it something else.

So I resumed the 100- and 200-house runs to 240,000 steps, which brings the 100-house run to the 45 episodes per house the 25-house run had.

The 200-house checkpoint at 240,000 steps became the final policy. Its name in the logs is h200cont, and everything from here uses it.

An arc is not a measurement

At this point the summary read “0.687 to 0.837”. Both numbers are real. The pair is not. 0.687 is one unseeded run from before the benchmark was made repeatable; the honest three-seed figure for imitation alone is 0.642. And 0.837 came from a run whose baseline arm was another RL checkpoint, not imitation. No paired measurement of imitation-alone against the full system existed on that benchmark. I had written the sections about unpaired reasoning, and then done it in the summary. The paired effects are what to quote. Two levels from two different runs are not.

14One look at the test split

By now the 147 validation episodes had been evaluated 204 times. Every decision had been made while looking at them. They were no longer held out in any honest sense. The 1,000 test houses had never been touched. I generated 147 episodes on 49 of them, with the same shape as validation, and measured once: imitation alone against the full system, three seeds, 441 paired trials.

0.626 against 0.889. +0.263, 66 wins against 2, p = 1.6 × 10^-17.

The effect is larger than anything on validation, and the failure table says why. Imitation wedged on 24% of test trials, against 15% on validation. These houses jam the imitation policy more, and the reflex removes every single one. Where there is more to fix, the fix is worth more. The system's own score, 0.889, is a little higher than on validation; imitation's is about the same. So this is not evidence the system is better than validation suggested. It is evidence about the houses.

And then the test split was spent. Any second measurement on it makes it a second look, and a second look is how validation stopped being held out. I did not touch it again.

15The wrong diagnosis, and a fix that made it worse

Time to look at what was left. On the 147 validation episodes: wedged 22 to 0. Wrong stops 19 to 4. Out of steps 5 to 20. Three of four failure modes were gone, and the fourth had grown. So: is the policy slow, or lost? I ran the same system with a 500-step budget instead of 250.

+0.014, p = 0.75. Doubling the time changed nothing. And at 500 steps, every remaining failure was an episode where the target was never once in view. Every episode where the robot saw the target eventually succeeded. So I concluded: the policy searches rooms that do not contain the target and never enters the one that does. The remaining problem is room choice. Knowing that fridges live in kitchens is semantic knowledge, and it sits in the frozen text encoder, where PPO cannot reach it.

That was a clean story, and I spent a week on it. I traced where the room knowledge went. A linear probe can read the target's room from the frozen text encoder's output with 0.814 accuracy. By the time that signal reaches the decoder, after the adapters and fusion, the probe reads 0.644. The knowledge was there and the trainable part was losing it.

So I fixed that. An auxiliary head on the decoder input, trained to predict the target's room, with a small loss weight, fine-tuned for three epochs against a control with the same schedule and no head.

The probe went from 0.644 to 0.844, the full value of the prior. Closed-loop success went from 0.6375 to 0.5864. -0.051, 77 wins against 128, p = 0.00045, all three seeds negative. The policy made 108 fewer wrong stops and ran out of steps 199 more times. It walked longer and committed less.

16The reflex removed the label, not the jam

So I stopped reading failure labels and recorded every action and every position of the full system over 200 episodes in 198 fresh houses.

A failing episode was not searching the wrong rooms. It was not searching. It covered about 5.5 square metres in 250 steps, and travelled 13 metres to do it: back and forth in one place. The action record says what kind of place.

137 of 250 actions in a failing episode do not succeed, against 1.3 of 37 in a success. The robot is wedged against geometry. It had been wedged all along.

Here is what the collision reflex had actually done. The “stuck” rule ends an episode after five failed moves in a row. The reflex forbids repeating the move that just failed. So the robot turns, tries, fails, turns back, tries, fails. Never five in a row. The counter never fires. The episode runs all 250 steps and is labelled “out of steps”. The reflex is still worth every point it measured; the robot does escape more often. But a lot of what it did was convert a jam into a timeout, and every later reading of the labels inherited that. “Stuck 22 to 0” and “out of steps 5 to 20” are the same fact.

The recovery reflex

The policy tries b in 1% of its actions in a failing episode. The expert's own recovery rule is: after two failed moves, back up twice. So, a third test-time rule. Count failed moves since the last successful one, turns not included. At two, force two steps of b. The policy still sees every frame; only the chosen action is overridden.

+0.022, 64 wins against 26, p = 0.000077, all three seeds positive. 0.841 to 0.863. Of the 81 episodes it won, 77 were timeouts for the baseline. It fixed exactly the failure the diagnostic named. Mean episode length went down, 67 to 63 steps: the robot escapes and then finds the target, rather than searching longer. And 215 of the 258 timeouts remain, so most of the jam is still there.

17The policy does not jam where it trained

A forced b is a patch. The policy never learned to recover. The recovery episodes in V8 start already jammed, with no history; the policy's real jams come after sixty-odd steps of ordinary driving, and look nothing like them. So the last idea: roll the policy out on its own training episodes, stop at the first jam, and let the expert drive from there. The saved episode then opens with a demonstrated escape from a jam the policy really produced. This is a cousin of DAgger, where the expert labels the learner's own states.

The first version found no jams. It judged a jam by the same counter the recovery reflex uses, and the collision reflex makes sure that counter never reaches its limit. The test I had just written about, defeating the test I had just written. So I changed it to a position test: at least six failed moves in a 25-step window, and the robot still within half a metre of where the window began.

The second version found no jams either, and that turned out to be the finding.

On 40 training episodes, the policy fails 0.1 moves per success and at most one per timeout. The timeouts are wandering, not wedging. On 40 episodes in unseen houses, two hit the jam test and two more fail 28 and 32 moves. The training data covers 9,918 of the 10,000 training houses, so “training house” means “a house the policy has seen”.

It also means the jam-capture idea cannot run on training data. There is nothing to capture. The only clean source of the policy's real jams is the 800 or so validation houses no split uses, and the expected yield is a few hundred expert episodes against 33,000. The one earlier attempt to teach a skill with targeted data, the 3,300 recovery episodes, did not remove the jams. I stopped the project here.

18Every idea, on one chart

The two biggest bars are not learning at all. They are fixing the question and fixing the teacher. Below them, RL and two small rules are about the same size as each other, and the rules were far cheaper to prove. Three ideas sit inside the noise band. One is clearly below zero, and it was the one with the best story.

The final system, for the record:

h200cont/best.ckpt          imitation policy, PPO fine-tuned across 200 houses, 240k steps
  + collision reflex        forbid the move that just failed
  + stop gate               no `end` below 50% probability or before step 5
  + recovery reflex         after 2 failed moves, force 2 steps back

held-out test, one look:    0.889 against 0.626 for imitation alone
594-episode dev split:      0.863 with the recovery reflex, 0.841 without

The two numbers do not combine. The test figure was measured before the recovery reflex existed, and the test split was spent, so the reflex was measured on the development split only.

19What I learned

1. Check that the number measures the task. The same policy scored 0.200 and 0.433 under two rules. Two weeks of conclusions came from the wrong one.
2. Demonstrations should sit inside the region they teach. Stopping the expert on the boundary of success gave 0.177. Stopping it well inside gave 0.633. The learner is always a little off.
3. You cannot learn an action the data never contains. The expert never reversed, so the policy never reversed, so it could not leave a door frame.
4. Offline metrics do not pick navigators. Macro F1 chose a 0.490 checkpoint over a 0.687 one. Evaluate closed-loop, and evaluate both checkpoints.
5. The agent optimises the rules as written. Spinning for free. Pressing on a wall because the straight line goes through it. Reversing across the house because backward collisions did not count. Every one was found in a video.
6. Seed everything, pair everything, report three seeds. An unseeded benchmark moved 0.08 with the policy unchanged. Every single-run comparison before that was noise.
7. Narrow beats broad in a paired test. A rule that changes few episodes, nearly all for the better, is easier to prove than RL that changes many episodes in both directions. Prefer interventions whose mechanism predicts which episodes will move.
8. Protect the actor from the critic. One detach() turned PPO from a thing that collapsed runs into a thing that gave +0.073.
9. When you sweep one knob, check what else moved. “More houses” was secretly “fewer visits per house”.
10. Labels are not behaviour. “Stuck” went to zero and “out of steps” went up by the same amount. The robot was doing the same thing under a different name. I built a week of work on the name.
11. A held-out split is spent after one look. Validation stopped being held out through 139 looks. Test was measured once and then closed.

20What stays open

The wedge in unseen houses. About one episode in ten in a new house ends with the robot rocking in place against geometry it cannot read. The recovery reflex removes a sixth of those. Nothing I have removes the rest. The policy has no collision sensor, no depth, and a frozen vision encoder, and I suspect all three matter.

Learning to recover, rather than being forced to. The jams only exist in unseen houses, so the clean source of examples is small. Whether a few hundred expert escapes teach a general skill is unknown. The 3,300 pre-jammed episodes did not.

Evaluation power. Every split has 147 to 594 episodes. Two questions, whether 25 houses beat 1 and whether 200 houses beat 100, failed on power rather than effect size. More seeds do not help when the seeds repeat the same episodes. More episodes would.

The original goal. This was meant to end with the policy navigating a scanned real room. It never got there. The scanned room is still untouched, which at least means it is still a clean test for whoever tries next.

Cite this post

If this write-up is useful in your work, you can cite it as a blog post:

@misc{kakooee2026phynav,
  author       = {Kakooee, Reza},
  title        = {Learning to Navigate, Failing to Recover},
  year         = {2026},
  howpublished = {\url{https://rezakakooee.github.io/phynav-objectnav/}},
  note         = {Blog post}
}