Enqueue default trial for hyperparam optimization

[araffin]

See https://optuna.readthedocs.io/en/latest/reference/generated/optuna.study.Study.html#optuna.study.Study

[caburu]

I've done this in my RL Zoo's fork. It would be very useful to have the validated implementation for that. I could open a PR* about this if it is of interest.

* In some weeks, after an article deadline :)

I've created methods in hyperparameters_opt.py to return the default parameters. Here for TD3 for example:

 def default_td3_params() -> Dict[str, Any]: 
     return { 
             "gamma": 0.99, 
             "lr": 0.001, 
             "batch_size": 100, 
             "buffer_size": 1000000, 
             "learning_starts": 100, 
             "tau": 0.005, 
             "episodic": True, 
             "noise_type": None, 
             "net_arch": "big",   
             } 

And a method to return the deafult parameters by algorithm's name:

 def default_params(algo: str) -> Dict[str, Any]: 
     if algo == "ppo": 
         return default_ppo_params() 
     elif algo == "sac": 
         return default_sac_params() 
     elif algo == "td3": 
         return default_td3_params() 
     else: 
         raise NotImplementedError("Missing default parameters for", algo, "algorithm!") 

Then I use it in exp_manager.py:

 try: 
     if self.default_param_as_first_trial: 
         # Include SB3 default params as the first trial. 
         study.enqueue_trial(default_params(self.algo)) 
                          
     study.optimize(self.objective, n_trials=self.n_trials, n_jobs=self.n_jobs) 

I have also created a command line argument in train.py to use this feature or not.

 parser.add_argument( 
     "--default-param-as-first-trial", help="Use algorithm's default parameters values as the first trial", action="store_true", default=False 
 ) 

[araffin]

Hello, I was more thinking about using the default hyperparams so you get for each env (already defined in the yaml files).

[caburu]

I was more thinking about using the default hyperparams so you get for each env (already defined in the yaml files).

Ok, I got it. For environments with default hyperparams already defined, yes, this would make more sense.

And, in fact, it would be a more general solution, because I believe that for the params not defined in the yaml files, the default library values would be used. So if I put a new env in the yaml file without define any params, my use case would be covered.