M2 icon indicating copy to clipboard operation
M2 copied to clipboard

floating point weirdness

Open DanGrayson opened this issue 8 years ago • 0 comments

This is probably something about precision used while printing.

i3 : 2^53

o3 = 9007199254740992

i8 : (2^53-1) + 0.

o8 = 9007199254740990

o8 : RR (of precision 53)

i10 : (2^53-2) + 0.

o10 = 9007199254740990

o10 : RR (of precision 53)

i11 : (2^53-3) + 0.

o11 = 9007199254740990

o11 : RR (of precision 53)

i12 : (2^53-4) + 0.

o12 = 9007199254740990

o12 : RR (of precision 53)

i13 : (2^53-5) + 0.

o13 = 9007199254740990

o13 : RR (of precision 53)

i14 : (2^53-6) + 0.

o14 = 9007199254740990

o14 : RR (of precision 53)

i15 : (2^53-7) + 0.

o15 = 9007199254740980

o15 : RR (of precision 53)



i21 : (2^53-1) + 0.

o21 = 9007199254740990

o21 : RR (of precision 53)

i22 : toExternalString oo

o22 = .9007199254740991p53e16

DanGrayson avatar Nov 17 '17 17:11 DanGrayson