mltrace icon indicating copy to clipboard operation
mltrace copied to clipboard

[EPIC] Abstractions for monitoring outputs

Open shreyashankar opened this issue 4 years ago • 0 comments

We want to monitor ML pipelines to see when things might be going wrong. We need a way to encode "SLAs" into mltrace, then monitor outputs and metrics to make sure SLAs are met.

To Dos:

Logging

  • [x] Add 2 new tables (outputs, feedback) that stores ts, key, val, identifiers/labels, indexed on ts, key
  • [x] Add functions to write and read from those tables

Querying

  • [x] Add function that joins the 2 tables
  • [x] #283

Interface

  • [x] Metric class with log_pred, log_true and compute_metric functions
  • [ ] "Monitor" view that accepts the group name(s), window size, and metric name
  • [ ] chartjs graph to show the above

shreyashankar avatar Oct 23 '21 02:10 shreyashankar