pudl
pudl copied to clipboard
Set up a remote experiment tracking server
In the process of setting up general record linkage infrastructure for the FERC - EIA match, we added experiment tracking infrastructure with MLflow that allows developers to track model parameters and results and with a local database that can be accessed with the MLflow UI.
This issue is to set up a remote tracking server for these experiments, so we can track metrics from all the models that are integrated into PUDL, and know when results change.
I think ideally we could be alerted if there is significant change in model results during a nightly build.
Nice To Haves (from experience using experiment tracking in its current state, not necessarily relevant to this issue):
- from Katie: it would be nice if from within Dagster you could launch a new experiment with a new experiment ID when you re-execute an op or re-execute from a certain step. Maybe you can already do this, but the only way I could figure out to start a new experiment was to start a fresh run by materializing an asset from the start.