oic
oic copied to clipboard
Many digit floating point numbers break scpi_parse_numeric()
When passed numeric arguments with more than ten significant digits, exponent_multiplier overflows. The mantissa is then multiplied by 1073441896 instead of the correct multiplier.
Changing exponent_multiplier from long to float fixes the issue, with only a trivial loss of precision.
Thanks for this. I'm a bit wary of switching to float, but then again I guess it is uncommon to need more than ten digits of precision. I'm a bit snowed under at the moment, but will have a look when I can to see if there is another solution that doesn't require floating-point, otherwise switch exponent_multiplier to float as you suggest.
Thanks for the respoinse.
I didn't choose 10 digits! I was using TestController to generate automated test values and it produced the 10 digit numbers. I found a function in TC to trim the number of digits, so all is well.