lucene
lucene copied to clipboard
Use Kahan summation for float aggregations to reduce errors
Description
With facet associations, we have the option of summing floats into an accumulator. This type of operation is prone to errors, which can be prevented by using summation algorithms specifically designed for floats, such as Kahan summation.
Neat -- I had never heard of Kahan summation. Here is its Wikipedia page.
Are we not already summing into intermediate double and then truncating to float at the end? Might be the easiest win, without performance hassles
Right now, the accumulators in FloatTaxonomyFacets
are floats. We keep a value for each ordinal, so we can end up using a lot of memory. With doubles, we would increase memory usage even more. Still, it's worth considering if that's better than the extra overhead from a fancy summation algorithm.