Default value for eval_horizon

[TimotheeMathieu]

Should the default for eval_horizon be 500 ?

[omardrwch]

I'd keep it as large as possible (as now) and put a time limit in the environment, if necessary. To avoid hiding these choices from the user.

[KohlerHECTOR]

Yes it should be 500 but could be change, it does not really matter.

[omardrwch]

Yes it should be 500 but could be change, it does not really matter.

Why should this be 500? @KohlerHECTOR

[TimotheeMathieu]

This should be 500 because 500 is the default for all control gym environment and is used in most benchmarks of control environments. This may be a deep rl thing. I think that there is no default in tabular rl so I think it is best to just go with the default that exists in deep rl.

[omardrwch]

If the gym environment has already a time limit (at 500), any eval_horizon > 500 will do the job. So I'd keep as large as possible by default. Some atari environments have pretty huge horizons (~30k).

[KohlerHECTOR]

@omardrwch Sorry for the authoritarian closing. I think indeed 500 is some kind of industry standard let us say. But in any case, this could be changed by the user when they code their experiments. Plus evaluation is pretty costly so on the contrary I would keep it as low as possible :) I guess in a dream world, we would have some config files with suggested values for n_steps n_evals eval_horizonfor different envs :)

[omardrwch]

No worries! Ok for 500, but then let's put warning if we've reached 500 and the episode is not terminated.