rpncalc icon indicating copy to clipboard operation
rpncalc copied to clipboard

Incorrect results for high precision division operations

Open kattelk opened this issue 1 year ago • 2 comments

I tried to solve 20 / (641 / 60) and calculated the part in brackets first: 641 enter 60 enter divide 20 enter switch divide

The result was 1.0, which is clearly not correct.

Is there a mistake here with the rounding?

I am using version 1.3.1.

kattelk avatar Jul 01 '23 13:07 kattelk

Interesting, this appears to be a bug with the dart statistics library I'm using.

Dividing an integer by a decimal with precision of 16 results in the results precision being truncated to 1.

Dividing an integer by a decimal with precision >16 results in a crash

I'll have to either find a workaround or submit a fix to the upstream library.

apesic avatar Jul 01 '23 21:07 apesic

A simple way to replicate this is

100 enter .1 ÷

In my case, it just freezes.

I'm also on 1.3.1

vilchesalves avatar Dec 11 '23 21:12 vilchesalves