ludwig icon indicating copy to clipboard operation
ludwig copied to clipboard

[Hyperopt] Experiment Management Integrations

Open ShreyaR opened this issue 3 years ago • 0 comments

Add integrations for experiment management for hyperopt.

Notes & Requirements:

  • Should work with a ray backend
  • The current tensorboard tfevents file only stores the tune logs, and not the actual trainer logs.
    • E.g. if AUROC is the metric being optimized, then tfevents only contains max(AUROC_val) for each step, instead of containing the AUROC_val metric directly.
    • Additionally, all metrics and losses for all splits should be logged.
  • Log additional parameters like learning rate, batch size, etc in the case of decay.
  • Log total number of parameters of the model as metadata
  • Should be able to do a parameter importance analysis.

ShreyaR avatar Jun 15 '22 23:06 ShreyaR