rpncalc
rpncalc copied to clipboard
Incorrect results for high precision division operations
I tried to solve 20 / (641 / 60) and calculated the part in brackets first: 641 enter 60 enter divide 20 enter switch divide
The result was 1.0, which is clearly not correct.
Is there a mistake here with the rounding?
I am using version 1.3.1.
Interesting, this appears to be a bug with the dart statistics library I'm using.
Dividing an integer by a decimal with precision of 16 results in the results precision being truncated to 1.
Dividing an integer by a decimal with precision >16 results in a crash
I'll have to either find a workaround or submit a fix to the upstream library.
A simple way to replicate this is
100 enter .1 ÷
In my case, it just freezes.
I'm also on 1.3.1