tempto icon indicating copy to clipboard operation
tempto copied to clipboard

Default tolerance for floats comparison should be deducted from current value

Open maciejgrzybek opened this issue 9 years ago • 0 comments

According to: https://github.com/prestodb/tempto/pull/96#discussion_r53013056 better default tolerance should be given. It's suggested to use tolerance equal to the difference between expected value of floating point number and the nearest representation of float. When user provides tolerance, that should be used. E.g. when user compares value 0.2, the next representable floating point value bigger than that is 0.20000001788139343262, the next representable floating point value smaller than 0.2 is 0.19999998807907104492 so we should pick tolerance as small as 0.0000000119209... (difference between 0.2 and the one representable floating point value closer to it). On the other hand, when comparing doubles (binary64), tolerance is different because next representable double after 0.2 is 0.20000000000000003886 and before 0.2: 0.19999999999999998335, so the tolerance in that case should be 2.77...E-17.

maciejgrzybek avatar Feb 22 '16 22:02 maciejgrzybek