TiddlyWikiFormula icon indicating copy to clipboard operation
TiddlyWikiFormula copied to clipboard

float math issues in 1.2.5

Open gMakaan opened this issue 3 years ago • 0 comments

Try this formulas without setting <<formulaPrecision>> and <<formulaFixed>>:

  • 0.99999 = 1
  • 0.999999 = 0.1 (wrong result at 6+ digits after point, but using <<formulaPrecision>> and <<formulaFixed>> can fix it)
  • 0.999999/1 = 0.1
  • trunc(1,1) = 0.9
  • trunc(1,5) = 0.99999
  • trunc(1,6) = 0.10 (with parameter 7 and more it returns same result 0.10)
  • trunc(-1,1) = -0.9000000000000001
  • trunc(-1,2) = -0.9900000000000005 (others in similar manner)
  • round(1,19) = 0.10
  • round(1,37) = 0.10 (here may be more such "wrong numbers")
  • round(-1,1) = -1.0000000000000002
  • round(-1,2) = -1.0000000000000004 (with parameter 3 and more last numbers changed via floating point precision errors)
  • round(-1,14) = -0.9999999999999958
  • round(-1,15) = -1.0000000000000002 (16, 17, 18 param value return -1 as it must)
  • round(-1,19) = -0.9999999999999999 (but it already out of bounds)

gMakaan avatar Jan 09 '22 17:01 gMakaan