MissionPlanner
MissionPlanner copied to clipboard
Decimal vs double
I've seen an issue where a cast from double to decimal causes an exception. Need to fix mavparam to remove all decimals
An int32 can be 100% represented by a double. So replace decimal with double. Only issue is with float rounding. So needs to be considered
Ie decimal.maxvalue < float.maxvalue
@CraigElder did you end up getting a tlog for this one?
@
can we assume that if the value is larger that decimal.maxvalue then we can ignore conversion issues and cast it directly to double ? my concerns : how can be a parameter value this small/large ? I suspects that flt_max is used somewhere... have to check if float.maxvalue to double then back to float will result the same number. if yes then it is easy, if not then it needs some more logic....
float a = 3.402823466e+38F;
float max_flt = Single.MaxValue;
double b = (double)a;
float c = (float)b;
Console.WriteLine("A :" + a);
Console.WriteLine("Max_flt :"+max_flt);
Console.WriteLine("B: "+b);
Console.WriteLine("C :"+c);
if (a == c) Console.WriteLine(" a == c");
if (a == max_flt) Console.WriteLine(" a == max_flt");
A :3.402823E+38
Max_flt :3.402823E+38
B: 3.40282346638529E+38
C :3.402823E+38
a == c
a == max_flt
The only remaining question is the first one: can we assume that if the value is larger that decimal.maxvalue then we can ignore conversion issues and cast it directly to double.