dentaku icon indicating copy to clipboard operation
dentaku copied to clipboard

ROUND does not work properly for inline floats

Open mensfeld opened this issue 5 years ago • 6 comments

Given rule like that: ROUND(1.23456789, 3) executing Dentaku like so:

Dentaku::Calculator.new.evaluate!('ROUND(1.23456789, 3)') #=> 0.1235e1

what is surprising is the argument is not an inline:

Dentaku::Calculator.new.evaluate!('ROUND(nr, 3)', nr: 1.2345678) #=> 1.235

mensfeld avatar Nov 28 '18 13:11 mensfeld

ROUNDUP and ROUNDDOWN work as expected.

mensfeld avatar Nov 28 '18 13:11 mensfeld

When parsing numeric literals, the value will either become an integer or a BigDecimal (see token_scanner.rb:94), and rounding a BigDecimal still returns a BigDecimal. On the other hand, context values are used as-is without any conversion, so the second result is still a float. ROUNDUP/DOWN use floor or ceil on the scaled value and then divide by a float, so they will always coerce to float.

Maybe it would be more consistent to always force numeric functions to return integer or BigDecimal?

rubysolo avatar Nov 29 '18 05:11 rubysolo

Maybe it would be more consistent to always force numeric functions to return integer or BigDecimal?

BigDecimal would be much better and much more natural imho.

mensfeld avatar Nov 29 '18 09:11 mensfeld

Not rushing but any changes here? :)

mensfeld avatar May 07 '19 17:05 mensfeld

Not so far, but I'll try to take a look at this soon.

rubysolo avatar May 21 '19 21:05 rubysolo

Thank you. Don't hesitate if you need any help!

mensfeld avatar May 22 '19 10:05 mensfeld