MicroDuck RL: From Simulation to the Real Robot — Part 10 of 14
This installment draws from the repository’s AGENTS.md, a collection of lessons the project team learned through many failed training runs. It is one of the most valuable parts of the series.
Watch Three Things
- The mean reward should be increasing.
- Episode length should match the task’s expectations. In a walking task, consistently short episodes usually mean the robot keeps falling.
- The reward term for the primary objective must actually improve. Total reward can rise entirely because of regularization terms while the desired skill never emerges. This is a common way to fool yourself: the curves look beautiful, but the robot stands still.
Every Penalty Term Must Be Zero or Negative
Episode_Reward/<term> records the weighted value. If a penalty function already returns a negative value and you give it a negative weight, the two signs cancel and create a positive reward. The policy will exploit that loophole aggressively. Whenever you write a configuration, stop and verify the sign convention.
A Zero-Weight Term Will Read Zero
Interpret each metric alongside the curriculum’s weight schedule. A penalty that appears inactive early in training may simply not have been enabled by the curriculum yet.
RL Will Exploit Every Loophole You Leave Open
If you do not define a required pose or constrain how a goal may be reached, the policy will find another way. As a rule of thumb, define critical behaviors with hard state gates—contact state, orientation thresholds, or latches—rather than small penalties. Penalties are soft constraints that a policy can trade off against other rewards. A gate is a hard constraint.
When Training “Fails,” Measure Before You Speculate
Load the checkpoint and run a headless evaluation across different spawn configurations and termination outcomes. Many apparent failures are actually caused by an early checkpoint or an evaluation mismatch, not by a policy that failed to learn.
Inspect Both Video and Metrics
A run can pass every metric and still look obviously wrong on video. Metrics are only projections of what you chose to measure; video shows the behavior itself.
What to Watch in TensorBoard
| Panel | What to check |
|---|---|
Episode_Reward/* |
Weighted reward terms; verify that the primary objective improves and every penalty remains ≤ 0 |
Episode_Termination/* |
Fall rate |
Curriculum/* |
Curriculum progress |
Metrics/* |
Tracking error |
The remaining installments are practical references: a command cheat sheet, a beginner’s practice roadmap, a hands-on test log and source map, and an FAQ.