mexprp icon indicating copy to clipboard operation
mexprp copied to clipboard

Input is always parsed as f64s, which can result in precision errors

Open IntrepidPig opened this issue 6 years ago • 0 comments

This is most noticeable when inputting decimals, for example:

eval::<ComplexRugRat>("2.2") // = becomes 2476979795053773/1125899906842624

However, it also happens when integers become large enough

eval::<ComplexRugRat>("200000000000000000000000") // = 199999999999999983222784

This can be fixed by parsing the number (in parse.rs) with the functions provided by rug. However this will require adding parse functions to the Num trait and passing the N generic type all the way down which will be quite a bit of work.

IntrepidPig avatar Apr 07 '18 00:04 IntrepidPig