pyyaml icon indicating copy to clipboard operation
pyyaml copied to clipboard

Incorrect rounding when loading base 60 floats

Open notcake opened this issue 2 years ago • 0 comments

When loading floats, PyYAML 6.0 correctly returns the nearest value for decimal floats, but not for base 60 floats:

Python 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.load( "60.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125", yaml.Loader)
60.00000000000001
>>> yaml.load("1:0.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125", yaml.Loader)
60.0 # should be 60.00000000000001

Background

Python floats are 64-bit IEEE 754 floats, with 53 bits of mantissa. The default rounding mode for parsing and arithmetic is rounding to the nearest value, with halfway values rounded so that the LSB is even.

Base 10

60.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125 is
    0b111100.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000_1
      ^---------------------53 bits------------------------^
Rounding up to the nearest value:
    0b111100.00000000000000000000000000000000000000000000001
which is 60.00000000000000710542735760100185871124267578125.

Base 60

But when loading a base 60 float, PyYAML converts the 1: and 0.000... parts to floats separately, then adds them together. ie. rounding happens twice, once when converting to floats and once when doing the addition.

0.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125 is
    0b000000.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000_1
                                                             ^----------------------53 bits----------------------^
Rounding down to an even LSB, since it is a halfway value:
    0b000000.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000
Adding 60:
    0b111100.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000
      ^----------------------53 bits-----------------------^
Rounding down to an even LSB, since it is a halfway value:
    0b111100.00000000000000000000000000000000000000000000000
which is exactly 60.0

notcake avatar May 02 '22 20:05 notcake