elementary icon indicating copy to clipboard operation
elementary copied to clipboard

[ELE-1934] Support anomalies on a custom SQL metric calculation

Open haritamar opened this issue 2 years ago • 0 comments

The goal here is to extend Elementary's anomaly detection by providing the ability to provide a custom SQL metric calculation, on top of which anomalies will be computed.

Considerations:

  • We want to allow a generic computation, and in particular allow the usage of more than one column in the computation.
  • Therefore it may make the most sense to define a new table test called custom_metric_anomalies
  • We'd also like to be able to extend it to support group by functionality (similar to the one mentioned here)

Suggested interface:

elementary.custom_metric_anomalies:
   metric_name: roi 
   sql: sum(revenue) / sum(cost)

General guidelines:

  • Implement a new test file called test_custom_metric_anomalies.sql, with similar structure to test_volume_anomalies.sql (under macros/edr/tests)
  • Extend table_monitoring_query.sql to include implementation of custom metric. In particular see the macro get_metric_query - we need to add implementation for the new metric there.
  • The test parameters above need to be addressed also in get_anomalies_test_configuration.sql:
    • metric_name should be added to test_configuration
    • sql needs to be added to metric_properties (this dictionary contains all parameters that affect the metric computation, and whose changes invalidate existing metrics)

NOTE - I didn't address how to surface metric_name later in the UI, requires some further thinking.

ELE-1934

haritamar avatar Oct 29 '23 23:10 haritamar