exp: hydra compose and dump

[dberenbaum]

Hydra can also be used to compose configuration from multiple files (see https://hydra.cc/docs/intro/#composition-example for an example). DVC can use Hydra to update params.yaml at runtime, providing more flexible project configuration.

Building off the previous example, assume you have multiple model types you want to train. You can specify parameters for each under conf:

$ tree
.
└── conf
    └── train
        ├── gradient_boosting.yaml
        └── random_forest.yaml

conf/train/gradient_boosting.yaml looks like:

algo: GradientBoostingClassifier # Specify model algorithm
seed: 20170428
n_est: 50
learning_rate: 0.1 # New parameter for gradient boosting

Now you can run a new gradient boosting experiment using that configuration:

$ dvc exp run --set-param +train=gradient_boosting

This will update the train section of params.yaml with the values from conf/train/gradient_boosting.yaml and run an experiment with the following params.yaml:

train:
  algo: GradientBoostingClassifier
  seed: 20170428
  n_est: 50
  learning_rate: 0.1

Originally posted by @dberenbaum in https://github.com/iterative/dvc/discussions/7044#discussioncomment-3271855