mltrace
mltrace copied to clipboard
[EPIC] Abstractions for monitoring outputs
We want to monitor ML pipelines to see when things might be going wrong. We need a way to encode "SLAs" into mltrace, then monitor outputs and metrics to make sure SLAs are met.
To Dos:
Logging
- [x] Add 2 new tables (outputs, feedback) that stores ts, key, val, identifiers/labels, indexed on ts, key
- [x] Add functions to write and read from those tables
Querying
- [x] Add function that joins the 2 tables
- [x] #283
Interface
- [x] Metric class with
log_pred,log_trueandcompute_metricfunctions - [ ] "Monitor" view that accepts the group name(s), window size, and metric name
- [ ] chartjs graph to show the above