kdb-studio icon indicating copy to clipboard operation
kdb-studio copied to clipboard

studio renders small floats inconsistently with q

Open adavies42 opened this issue 2 years ago • 1 comments

studio rounds very small floats to zero, which q never does

e.g. enter 7.275958e-12 in studio and run it, and it returns 0f, while the q console would say 7.275958e-12 for the same input

adavies42 avatar Jul 24 '23 20:07 adavies42

Apparently q switches to exponential notation if the exponent is less than -4 or greater than 16. Do we want to mirror this behavior in studio? Or should it be configurable, like there is already a "decimal digits" setting? Using that setting it's possible to view this number as non-zero, although it still uses the full decimal notation, e.g. setting it to 20 gives 0.000000000007275958f.

gyorokpeter avatar Sep 26 '23 11:09 gyorokpeter