engine icon indicating copy to clipboard operation
engine copied to clipboard

fix: mathematical precision

Open cptbtptpbcptdtptp opened this issue 2 years ago • 2 comments

The number in JS uses the IEEE754 standard: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number

IEEE754 conversion rules: https://www.h-schmidt.net/FloatConverter/IEEE754.html

$2^{53}\rightarrow 10^{15.95}$

cptbtptpbcptdtptp avatar Aug 10 '22 04:08 cptbtptpbcptdtptp

Encoding is: $2^{\left( 2-v0\right) }\times \left( 1+v1\right)$ Maybe is $2^{52}-1\rightarrow 10^{15.65}$ Math.floor(15.653559774527022)=15 ?

GuoLei1990 avatar Aug 10 '22 06:08 GuoLei1990

Should fix Math unit test

GuoLei1990 avatar Aug 10 '22 09:08 GuoLei1990