Add CI/CD Model Runs

[jacobbieker]

trafficstars

Detailed Description

To know how changes are affecting model performance, it could be nice to be able to run a model with commands from PRs and record how it compares to the baseline/other models

GitHub has some actions that can be accomplished to do that: https://github.blog/2020-06-17-using-github-actions-for-mlops-data-science/

Context

We need to compare our models to other PV predictions, and so this could be a nice way to record some model performance, in addition to Neptune.ai

Possible Implementation

https://github.blog/2020-06-17-using-github-actions-for-mlops-data-science/ https://github.com/machine-learning-apps/actions-ml-cicd

[dudeperf3ct]

@jacobbieker I would be interested in submitting a PR!

Do we have a workflow in the works? e.g. from above blog would be using chatops gh action to communicate using pr, some gh action to submit (Argo or CML) ML workflow (we have to select which models to run against by modifying config.yaml or some kind of matrix) in addition to logging to Neptune and display the metrics and plots on the pr for ease of comparison.

Some questions like do we trigger this workflow by making some changes to default parameters or simply run with default parameters for selected models for every trigger?

[jacobbieker]

Hi! We currently don't have a workflow for this, and there isn't one in the works. I think the ideal way would be to trigger it based off when a model or datamodule config changes. We've used CML before for Satip.