MissionPlanner icon indicating copy to clipboard operation
MissionPlanner copied to clipboard

Decimal vs double

Open meee1 opened this issue 1 year ago • 5 comments

I've seen an issue where a cast from double to decimal causes an exception. Need to fix mavparam to remove all decimals

An int32 can be 100% represented by a double. So replace decimal with double. Only issue is with float rounding. So needs to be considered

Ie decimal.maxvalue < float.maxvalue

meee1 avatar Jan 03 '24 21:01 meee1

@CraigElder did you end up getting a tlog for this one?

meee1 avatar Jan 03 '24 21:01 meee1

image_2023_12_13T20_41_33_948Z-1.png

meee1 avatar Jan 03 '24 21:01 meee1

@1702500308897-1.JPEG

meee1 avatar Jan 03 '24 21:01 meee1

can we assume that if the value is larger that decimal.maxvalue then we can ignore conversion issues and cast it directly to double ? my concerns : how can be a parameter value this small/large ? I suspects that flt_max is used somewhere... have to check if float.maxvalue to double then back to float will result the same number. if yes then it is easy, if not then it needs some more logic....

EosBandi avatar Jan 03 '24 23:01 EosBandi

float a = 	3.402823466e+38F;
float max_flt = Single.MaxValue;
double b = (double)a;
float c = (float)b;
Console.WriteLine("A :" + a);
Console.WriteLine("Max_flt :"+max_flt);
Console.WriteLine("B: "+b);
Console.WriteLine("C :"+c);
if (a == c) Console.WriteLine(" a == c");
if (a == max_flt) Console.WriteLine(" a == max_flt");
A :3.402823E+38
Max_flt :3.402823E+38
B: 3.40282346638529E+38
C :3.402823E+38
 a == c
 a == max_flt

The only remaining question is the first one: can we assume that if the value is larger that decimal.maxvalue then we can ignore conversion issues and cast it directly to double.

EosBandi avatar Jan 04 '24 17:01 EosBandi