Four controllers learned by reinforcement on the lab's simulator, each on a different recipe: the model exactly as written, or the model with its doubts drawn afresh every episode, over a narrow range, a wide one, or the shipped one with a memory. Today's machine is not the model: its motor answers late, its track tilts, its top link weighs what it weighs. Watch the model's own controller cope, pick a recipe, and take Kick's ten kicks. Then the other three show what they would have done.
The test is Kick's: ten tip kicks, 0.4 to 1.9 rad/s, one every six seconds; a kick is survived if the machine still stands when the next is due; cost is ten per kick lost, five times the control energy, and half the cart's wandering while it stands. The four recipes are real files, the ones in games/policies/: the model alone and a narrow range are balance policies (soft actor-critic), handed the machine where the lab's planned catch would and returned to the planned stages when they fall; a wide range and the range, with a memory do every stage themselves and get up on their own. Before you run, the machine is held by the lab's planned controller, which knows the model and nothing of today; its wobble and its falls are your only reading of how far today is from the model. The ladder stays hidden until your first run of the day, and then shows all four, and the planned controller beside them. Hidden today: the delay, up to 40 ms; a tilt, up to 2 N; the top link's mass, ×0.75 to ×1.35.
A policy trained on the model as written learns the model's every quirk, and on a day when the world is the model it is the best thing here: precise, cheap, and ten for ten. Move the world a few milliseconds and it has never seen the situation and falls without warning. Draw the model's doubtful numbers afresh every episode and the policy can no longer memorise any one world; it learns the family instead, at the price of precision on each member, and the wider the family the higher the price. So the range is a bet: narrow if you trust the model, wide if you do not, and a memory in the input if you want the policy to place today's world within the family on its own. Nobody tells you the world. That is the whole of sim-to-real in one dial, and the dial is set before anything is learned, by someone guessing how wrong the model is. Some days the guess is wrong in the other direction, and the ladder says so.
Every simulator is a model, and a model omits things: a delay, a friction, a link's true mass, a sensor's noise. A policy that has only ever seen the model's world has no reason to work outside it, and reinforcement learning is very good at exploiting exactly the things a model gets wrong. The reality gap is the name for what happens next. The remedies are all ways of not letting the policy trust the model: randomise the doubtful numbers, add noise, learn from real data too, or give the policy a way to tell where it is, as the memory does here.
The honest measure is not the training curve but a test on a machine the training never saw, which is what today's world is. Every controller in this series is designed on a model and run on a world that differs; this page only makes the design step a learned one, and the difference is that the model-based controllers can be re-derived when the model changes, and these cannot.
Measured here on Kick's ladder, across the days: the model-alone policy takes ten kicks of ten when the delay is under ten milliseconds and none when it is over twenty; the narrow-range policy holds seven or eight to about 25 ms and then drops away; the wide one keeps five on most days and rarely more; the one with the memory takes the harsh days and gives back a rung on the mild ones. No recipe wins every day, and the ladder shows the day. Automatic domain randomisation widens the range as the policy masters it, which is one way to stop guessing; a memory in the input, or an explicit estimate of the world fed to the policy, is the other, and it is Identify's idea done by a network.
The lesson generalises past learning. A controller designed to be robust over a set is worse on every member of the set than one designed for that member; robust control is the theory of how much worse, and identification is the way to shrink the set. The recipe dial is that trade with a training run behind it.