ludwig
ludwig copied to clipboard
[Hyperopt] Experiment Management Integrations
Add integrations for experiment management for hyperopt.
Notes & Requirements:
- Should work with a ray backend
- The current tensorboard
tfeventsfile only stores the tune logs, and not the actual trainer logs.- E.g. if AUROC is the metric being optimized, then
tfeventsonly containsmax(AUROC_val)for each step, instead of containing theAUROC_valmetric directly. - Additionally, all metrics and losses for all splits should be logged.
- E.g. if AUROC is the metric being optimized, then
- Log additional parameters like learning rate, batch size, etc in the case of decay.
- Log total number of parameters of the model as metadata
- Should be able to do a parameter importance analysis.