tdigest icon indicating copy to clipboard operation
tdigest copied to clipboard

WIP: experimental building of histograms from tdigest

Open tvondra opened this issue 4 years ago • 1 comments

This implements two simple SQL functions, building different types of histograms from a t-digest. Equi-width histograms are the "usual" type with each bin covering about 1/N of the domain, while equi-height histograms have bins representing the same fraction of the data.

The functions are fairly trivial SQL functions, calculating boundaries on either the domain or frequency, and then computing the other values using the t-digest.

I'm not sure how accurate the histograms are - considering how t-digests work, the bins at the ends should be the fine while the bins in the middle could be somewhat less accurate.

One thing this made me realize is the extension only provides aggregate functions, so when we need to work with t-digests in non-aggregate contexts it's necessary to do unnecessary groupings etc. Might be handy to provide some non-aggregate functions to make that unnecessary.

tvondra avatar Aug 09 '20 19:08 tvondra

These look pretty good to me at first glance, and from my testing of them. I will spend some more time thinking about how I would want to build out histograms for my own usage and how this all relates. Thank you!

sporty81 avatar Aug 10 '20 16:08 sporty81