GodotBigNumberClass icon indicating copy to clipboard operation
GodotBigNumberClass copied to clipboard

roundDown() not quite working as expected

Open Leux022 opened this issue 1 year ago • 2 comments

Ex: Big.new(275.75).roundDown().toString() giving 276

I think it's because snapped(mantissa, precision) is rounding to the nearest multiple of precision, not down.

I'm not sure if it's actually a good fix but I replaced mantissa = snapped(mantissa, precision) with mantissa = floor(mantissa * (1/precision)) * precision

Apologies for formatting and/or if I wasn't supposed to submit an issue about this, still learning github.

Leux022 avatar Dec 21 '23 14:12 Leux022

Edit-- my original reply said your method doesn't work. My toScientific() method was rounding the text. Looks like your method is good!

Side note: I asked AI for help, and it came up with, instead of your solution, mantissa = floor(mantissa / precision) * precision. So it just swapped the order of multiplication and division. Kinda funny!

Drillur avatar Jan 08 '24 01:01 Drillur

I'm not sure if it's actually a good fix but I replaced mantissa = snapped(mantissa, precision) with mantissa = floor(mantissa * (1/precision)) * precision

Did mantissa = floor(mantissa / precision) * precision instead, since the 1 / precision is unnecessary: 74076d2

TehAwesomestKitteh avatar Jan 31 '24 03:01 TehAwesomestKitteh