monosi icon indicating copy to clipboard operation
monosi copied to clipboard

Fix standard deviation metrics

Open unkrich opened this issue 2 years ago • 0 comments

Description

Numeric standard deviation and length standard deviation are not currently being calculated

https://github.com/monosidev/monosi/blob/master/src/ingestion/sources/base.py#L265-L275

Expected behavior

By uncommenting these lines, we would expect that the metrics are made available to users. It would seem that the SQL being run in this case is not correctly computing the standard deviation of length and values.

The current SQL is here:

  • https://github.com/monosidev/monosi/blob/master/src/ingestion/sources/base.py#L451
  • https://github.com/monosidev/monosi/blob/master/src/ingestion/sources/base.py#L467

Steps to reproduce

  1. Uncomment https://github.com/monosidev/monosi/blob/master/src/ingestion/sources/base.py#L265-L275 to enable the metrics. 2. Then try to run the application.
  2. You'll notice that queries fail due to the metric being calculated incorrectly. There is also the potential that it works and there is some specific niche case where this was an issue, so this should be verified.

Additional context

To fix this issue, one simply needs to figure out the SQL required to calculate the numeric std dev and length std dev and replace the fields here. Then, uncomment the metric lines.

unkrich avatar Apr 13 '22 18:04 unkrich