math icon indicating copy to clipboard operation
math copied to clipboard

RFC: add `getPrecision` method.

Open bendavies opened this issue 3 years ago • 11 comments

Hi,

Similar to getScale, would you be on favor of adding a method getPrecision? I want to perform app side validation on BigDecimal before passing to Postgres numeric fields, and getPrecision would make that easier.

Thanks!

bendavies avatar May 19 '21 08:05 bendavies

Hi, what would be the difference between getScale() and getPrecision()?

BenMorel avatar May 19 '21 21:05 BenMorel

hi @BenMorel

The PostgreSQL manual will say it better than me:

We use the following terms below: The precision of a numeric is the total count of significant digits in the whole number, that is, the number of digits to both sides of the decimal point. The scale of a numeric is the count of decimal digits in the fractional part, to the right of the decimal point. So the number 23.5141 has a precision of 6 and a scale of 4. Integers can be considered to have a scale of zero.

bendavies avatar May 20 '21 08:05 bendavies

@BenMorel any interest? thanks!

bendavies avatar Sep 30 '21 12:09 bendavies

@BenMorel

Working with precision is generally better than working with scale

Check this example:

(0.03/360)*8*30 = 0.02

Woking with scale of 10: 0.03/360 = 0.0000833333(3) 0.0000833333*8=0.0006666664(0) 0.0006666664*30=0.0199999920(0)

❌ 0.0199999920 == 0.02

Working with precision of 10: 0.03/360 = 0.00008333333333(3) 0.00008333333333*8=0.0006666666666(40) 0.0006666666666*30=0.01999999999(80)=0.02

✔0.02==0.02

alexviar avatar Jul 30 '22 03:07 alexviar

@alexviar i'm not sure what your example shows but it's not precision of 10

Working with precision of 10: 0.03/360 = 0.00008333333333(3)

0.00008333333333(3) has as precision of 16

bendavies avatar Aug 02 '22 10:08 bendavies

@alexviar i'm not sure what your example shows but it's not precision of 10

Working with precision of 10: 0.03/360 = 0.00008333333333(3)

0.00008333333333(3) has as precision of 16

It has a scale of 14 but a precision of 10. The precision refers to how many significant digits it has, not how many decimal places it has. https://www.logicbig.com/quick-info/programming/precision-and-scale.html

For example Big.js uses scale whereas Decimal.js uses precision

alexviar avatar Aug 03 '22 16:08 alexviar

@alexviar i'm not sure what your example shows but it's not precision of 10

Working with precision of 10: 0.03/360 = 0.00008333333333(3)

0.00008333333333(3) has as precision of 16

I did put in parentheses digits that was rounded, and underlined digits are the Periodo of decimal

alexviar avatar Aug 03 '22 17:08 alexviar

ah right thanks, i forgot that leading zeros are not significant. but also for storage purposes (postgres), 0 <= scale <= precision so, It has a scale of 14 but a precision of 10. doesn't work, in practise.

anyway, the point of this of was to simple add a getPrecision() method, similar to getScale, nothing mnore

bendavies avatar Aug 04 '22 10:08 bendavies

Hi @bendavies, sorry for the late reply. I'm not against adding a getPrecision() method if it is useful to you, so please feel free to open a PR along with a few tests, in particular to clarify what happens for leading zeros.

@alexviar I'm not sure what the decimal period has to do with the precision? Anyway, you make me think that this would be a great addition to BigRational: a way to output 10/3 as 3.3(3) for example. Would you want to work on this?

BenMorel avatar Aug 04 '22 23:08 BenMorel

I opened issue #74 to track the BigRational as decimal number with period separately.

BenMorel avatar Aug 04 '22 23:08 BenMorel

@alexviar I'm not sure what the decimal period has to do with the precision?

I think I didn't explain myself well. I did put rounded decimals (and decimal period) in parentheses for illustrative purposes only. Precision is given by working with n significant digits instead of n decimal place:

Operation Rounded to 10 significant digits Rounded to 10 decimal places
0.03 0.03000000000 0.0300000000
360 360.0000000 360.0000000000
8 8.000000000 8.0000000000
30 30.00000000 8.0000000000
0.03/360 0.00008333333333 0.0000833333
(0.03/360)*8 0.0006666666666 0.0006666666
(0.03/360)*8*30 0.02000000000 0.0200000000

Now, if we take the rounded values to do subsequent operations, we have:

Operation Rounded to 10 significant digits Rounded to 10 decimal places
0.03 0.03000000000 0.0300000000
360 360.0000000 360.0000000000
8 8.000000000 8.0000000000
30 30.00000000 8.0000000000
0.03/360 0.00008333333333 0.0000833333
(0.03/360)*8 0.0006666666666 0.0006666664
(0.03/360)*8*30 0.02000000000 0.0199999920

Currently it can be circunvent by using BigRational, but this workaround doesn't work with irrational opreations like square root for example (~~sqrt(2)*sqrt(2)~~ Never mind, I had read an article with an example like this, but I can not found it)

alexviar avatar Aug 06 '22 00:08 alexviar