pyyaml
pyyaml copied to clipboard
Incorrect rounding when loading base 60 floats
When loading floats, PyYAML 6.0 correctly returns the nearest value for decimal floats, but not for base 60 floats:
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.load( "60.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125", yaml.Loader)
60.00000000000001
>>> yaml.load("1:0.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125", yaml.Loader)
60.0 # should be 60.00000000000001
Background
Python floats are 64-bit IEEE 754 floats, with 53 bits of mantissa. The default rounding mode for parsing and arithmetic is rounding to the nearest value, with halfway values rounded so that the LSB is even.
Base 10
60.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125 is
0b111100.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000_1
^---------------------53 bits------------------------^
Rounding up to the nearest value:
0b111100.00000000000000000000000000000000000000000000001
which is 60.00000000000000710542735760100185871124267578125.
Base 60
But when loading a base 60 float, PyYAML converts the 1:
and 0.000...
parts to floats separately, then adds them together.
ie. rounding happens twice, once when converting to floats and once when doing the addition.
0.00000000000000355271367880050132378607394839652770586428264139311483660321755451150238513946533203125 is
0b000000.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000_1
^----------------------53 bits----------------------^
Rounding down to an even LSB, since it is a halfway value:
0b000000.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000
Adding 60:
0b111100.00000000000000000000000000000000000000000000000_10000000000000000000000000000000000000000000000000000
^----------------------53 bits-----------------------^
Rounding down to an even LSB, since it is a halfway value:
0b111100.00000000000000000000000000000000000000000000000
which is exactly 60.0