dmcontrol-generalization-benchmark

dmcontrol-generalization-benchmark copied to clipboard

The number of updates to the actor network, SAC's alpha value, and critic_target networks was too many (with default arg values) during the for-loop when step == args.init_steps. This is because "step" was given as an argument to the agent.update() function, instead of the index of the for loop. This argument is used on line 147 and 150 of SAC: https://github.com/nicklashansen/dmcontrol-generalization-benchmark/blob/a81df68838757ca55c19ac5f9fd8cd04bf8a942c/src/algorithms/sac.py#L147C1-L147C1

Depending on the value of args.init_steps and args.actor_update_freq this means that the actor network was either updated 0 times or args.init_steps times. (For default args values: the actor network was updated 1000 times, while it should only be updated 500 times, as args.actor_update_freq==2)

Good catch. I believe this would be a cleaner fix?

if step >= args.init_steps:
  num_updates = args.init_steps if step == args.init_steps else 1
  for i in range(num_updates):
    agent.update(replay_buffer, L, step+i)

Adapted from https://github.com/nicklashansen/tdmpc/blob/b8e945ed4368084e2a370dd239d668e3d9d95bf3/src/train.py#L70 which does seem to account for this.