Q-Operating-System icon indicating copy to clipboard operation
Q-Operating-System copied to clipboard

calc.c 2+0.1 = 2.99999991

Open stnKrisna opened this issue 10 years ago • 6 comments

Calculating n+0.1 returns n.99999991. Calculating n+0.2 returns n.19999982 (not that far off). Its just weird

stnKrisna avatar Dec 01 '15 12:12 stnKrisna

Could you also check more than one decimal placed values? Just rechecked my code and think there might be an issue for more values like 0.23 evaluated as 0.5...

plankp avatar Dec 01 '15 16:12 plankp

I'll check this later. PC and laptop (VM) are on a scheduled maintenance (virus scan and malware scan) and its 12:16 AM in my country, plus I have class at 9 later. I have to sleep first.

stnKrisna avatar Dec 01 '15 17:12 stnKrisna

So, I accidentally typed 0+0.25 and the result it returns is 0.7. And 2+0.25 returns 2.70000008. So, I think it has something to do with floating point rounding (tested with VMware Fusion)

stnKrisna avatar Dec 02 '15 09:12 stnKrisna

No 0.23 -> 0.5 is due to the code I wrote...

plankp avatar Dec 02 '15 11:12 plankp

A pure program error... (Fixed the 0.23 -> 0.5 issue already)

plankp avatar Dec 02 '15 11:12 plankp

This issue ism most likely caused by floating point conversions. We need a double to string method...

plankp avatar Dec 06 '15 16:12 plankp