scientific icon indicating copy to clipboard operation
scientific copied to clipboard

Integer overflow in exponent

Open ksvanhorn opened this issue 6 years ago • 3 comments

scientific 1 (maxBound::Int) * scientific 1 (maxBound::Int) evaluates to 1.0e-2.

ksvanhorn avatar Apr 17 '18 21:04 ksvanhorn

Yes, this is unfortunate. Scientific uses an Int to represent the exponent so it suffers from the same integer overflows that Int suffers from. I've an old TODO to switch the exponent to Integer to make Scientific truly arbitrary precision.

basvandijk avatar Apr 17 '18 22:04 basvandijk

I started doing some work on this on the integer-exponent branch.

basvandijk avatar Apr 17 '18 23:04 basvandijk

I'd prefer we throw an impure error instead of producing wrong values, till we can get arbitrary precision would a patch be accepted that calls Prelude(error) on any operation that would result in corruption?

mbj avatar Jul 30 '22 02:07 mbj