CleanRL is a deep reinforcement learning library (released last year), which allows users to quickly implement deep RL algorithms to a variety of environments. Each algorithm is written in a single Python file, which makes it easier to modify and even understand algorithms. The library can be found at [ Ссылка ]

The environment used here is Humanoid-v2 in MuJoCo via OpenAI Gym. The 17-DOF humanoid is rewarded for moving forward as quickly as possible, and learns to walk from scratch. Simulation episodes end automatically when the humanoid falls or when the humanoid hasn't fallen within 1000 timesteps.

The PPO algorithm did not converge to a walking behavior within 20 million timesteps. The SAC algorithm is known for being more sample efficient, but it took a long time to get to just 1.9M simulation timesteps, so I just stopped it there. Although it hadn't converged to a walking behavior yet, it was pretty far along in a relatively short simulation period. The TD3 algorithm was quicker than SAC, and it converged to a walking behavior. This doesn't make it the best of the three algorithms, but it didn't nearly take as long as SAC to reach 1-2M timesteps, and actually converged to a walking behavior in less than 20M timesteps (unlike PPO).

свернуть