pudl icon indicating copy to clipboard operation
pudl copied to clipboard

Set up a remote experiment tracking server

Open katie-lamb opened this issue 1 year ago • 0 comments

In the process of setting up general record linkage infrastructure for the FERC - EIA match, we added experiment tracking infrastructure with MLflow that allows developers to track model parameters and results and with a local database that can be accessed with the MLflow UI.

This issue is to set up a remote tracking server for these experiments, so we can track metrics from all the models that are integrated into PUDL, and know when results change.

I think ideally we could be alerted if there is significant change in model results during a nightly build.

Nice To Haves (from experience using experiment tracking in its current state, not necessarily relevant to this issue):

  • from Katie: it would be nice if from within Dagster you could launch a new experiment with a new experiment ID when you re-execute an op or re-execute from a certain step. Maybe you can already do this, but the only way I could figure out to start a new experiment was to start a fresh run by materializing an asset from the start.

katie-lamb avatar Feb 16 '24 18:02 katie-lamb