MicroDuck RL: From Simulation to the Real Robot — Part 04 of 14
With the environment ready, it is time to start training. This part explains every important parameter behind the one-line command.
Start Training
uv run train Mjlab-Velocity-Flat-MicroDuck \
--env.scene.num-envs 4096 \
--agent.logger tensorboard
--env.scene.num-envs 4096creates 4,096 parallel simulation environments. This is the key to throughput: every PPO update collects 4,096 × 24 = 98,304 environment steps.--agent.logger tensorboardselects TensorBoard. The default logger is Weights & Biases, which requires an account and sign-in. If you do not use W&B, switch to TensorBoard. Logs are written locally tologs/rsl_rl/velocity/<timestamp>_velocity/with no loss of functionality.
Important Defaults
Run uv run train <TASK> --help to see every available parameter. The most important defaults are:
| Parameter | Default | Meaning |
|---|---|---|
| Physics timestep | 0.005 s | MuJoCo advances by 5 ms per physics step |
| Decimation | 4 | One control action every four physics steps: 0.02 s, or 50 Hz |
| Episode length | 20 s | Each environment resets every 20 seconds |
--agent.max-iterations |
50,000 | Maximum number of training iterations |
--agent.save-interval |
250 | Saves a model_*.pt checkpoint every 250 iterations |
| Learning rate | 1e-3, adaptive | Adjusted automatically based on KL divergence |
| Epochs / minibatches | 5 / 4 | PPO update structure for each batch of experience |
What Happens in One Iteration
- Each of the 4,096 environments advances 24 steps, with policy inference running in parallel on the GPU.
- The system computes rewards, termination conditions, and generalized advantage estimates (GAE) for every step.
- PPO updates the policy using 5 epochs and 4 minibatches.
- The logger writes TensorBoard data, and a checkpoint is saved every 250 iterations.
Measured Performance on an RTX 5080
| Metric | Result |
|---|---|
| Time per iteration | Approximately 1.13 s |
| VRAM usage | Approximately 7.3 GB of 16 GB |
| GPU utilization | Approximately 84% |
| Rule-of-thumb training budget | Roughly 1,000 iterations for simple skill tasks; 4,000–6,000 for locomotion tasks, based on the project’s experience notes |
Resume from a Checkpoint
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096 \
--agent.run-name resume \
--agent.load-checkpoint model_2000.pt \
--agent.resume True
Run a Smoke Test First
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 64 --agent.max-iterations 5
According to the project documentation, a smoke test with 64 environments and 5 iterations catches roughly 95% of configuration errors at almost no cost. Always run it before committing to a long training job.
Part 05 moves into the most technical part of the series: a detailed look at the task’s observations, rewards, domain randomization, and curriculum learning.