Allow self-tuning submissions to specify `dropout_rate` and `aux_dropout_rate`

[Niccolo-Ajroldi]

Original issue

Self-tuning submissions use default values for dropout_rate and aux_dropout_rate (see #753). We want to allow them to specify custom values for these hyperparameters.

Solution

Allow self-tuning submission to provide a JSON file specifying values for these hyperparameters. The JSON file should look like this:

{
    "dropout_rate": 0.1,
    "aux_dropout_rate": 0.2,
}

This is passed to submission_runner.py through --tuning_search_space.

We read from such JSON file, and check that:

1. it specifies a value only for the allowed hyperparameters, and that
2. it specifies one single value for each of them.

We then proceed to call train_once with the correspondent fixed hyperparameters.

Discussion

I think this solution is pretty clean, I am just dubious about passing the JSON file path trough tuning_search_space., as this might create some confusion. Perhaps we shall change the name of this flag, or we can add a different flag?

Fix #753

[github-actions[bot]]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[priyakasimbeg]

I agree with you, I think we should add an additional flag that is optional for self-tuning and disallowed for external-tuning ruleset (so raise an error if the value of the flag is not none for external-tuning). Perhaps we can call this flag self_tuning_init_hyperparemeters_config? Maybe that is a bit long, do you have any suggestions for names?

[priyakasimbeg]

Underlying issues addressed in of https://github.com/mlcommons/algorithmic-efficiency/pull/875