stringencoders icon indicating copy to clipboard operation
stringencoders copied to clipboard

Round-to-even bug in modp_dtoa2

Open jeroen opened this issue 5 years ago • 0 comments

We found that modp_dtoa2 has odd cases where it does not properly round-to-even. For example x.665 should be rounded down to x.66 but sometimes it gets rounded to x.67

modp_dtoa2(9.8665, digits = 3)
# 9.867
modp_dtoa2(1.8665, digits = 3)
# 1.866

Or

modp_dtoa2(111.665, digits = 2)
# 111.67
modp_dtoa2(222.665, digits = 2)
# 222.66 
modp_dtoa2(333.665, digits = 2)
# 333.67
modp_dtoa2(444.665, digits = 2)
# 444.67

jeroen avatar Dec 27 '19 00:12 jeroen