oic icon indicating copy to clipboard operation
oic copied to clipboard

Many digit floating point numbers break scpi_parse_numeric()

Open palmerr23 opened this issue 3 years ago • 2 comments

When passed numeric arguments with more than ten significant digits, exponent_multiplier overflows. The mantissa is then multiplied by 1073441896 instead of the correct multiplier.

Changing exponent_multiplier from long to float fixes the issue, with only a trivial loss of precision.

palmerr23 avatar Jan 01 '22 05:01 palmerr23

Thanks for this. I'm a bit wary of switching to float, but then again I guess it is uncommon to need more than ten digits of precision. I'm a bit snowed under at the moment, but will have a look when I can to see if there is another solution that doesn't require floating-point, otherwise switch exponent_multiplier to float as you suggest.

LachlanGunn avatar Jan 20 '22 09:01 LachlanGunn

Thanks for the respoinse.

I didn't choose 10 digits! I was using TestController to generate automated test values and it produced the 10 digit numbers. I found a function in TC to trim the number of digits, so all is well.

palmerr23 avatar Jan 20 '22 22:01 palmerr23