unitwise icon indicating copy to clipboard operation
unitwise copied to clipboard

Conversion of ml/h to itself yields floating point/rounding inaccuracy

Open floriandejonckheere opened this issue 3 years ago • 1 comments

I've noticed that for some values of mL/h, converting to itself yields an inaccurate result:

Unitwise(4, "mL/h").convert_to("mL/h")
=> #<Unitwise::Measurement value=3.999999993 unit=mL/h>

Oddly enough, this doesn't happen for all values, e.g. 0-3, 5 and 10 yield the correct result. And it seems that mL/h is the only unit where this happens: L, dL, hL, s, min, h in any variation seem to be correct:

[:mL, :L, :dL, :hL]
  .product([:ms, :s, :min, :h])
  .map { |n, d| [n, d].join("/") }
  .index_with { |unit| (0..1000).select { |i| Unitwise(i, unit).convert_to(unit).value != i }.count }

=> {"mL/ms"=>0,
 "mL/s"=>0,
 "mL/min"=>0,
 "mL/h"=>981,
 "L/ms"=>0,
 "L/s"=>0,
 "L/min"=>0,
 "L/h"=>0,
 "dL/ms"=>0,
 "dL/s"=>0,
 "dL/min"=>0,
 "dL/h"=>0,
 "hL/ms"=>0,
 "hL/s"=>0,
 "hL/min"=>0,
 "hL/h"=>0}

floriandejonckheere avatar Apr 29 '21 12:04 floriandejonckheere

The following diff appears to address this:

diff --git a/lib/unitwise/unit.rb b/lib/unitwise/unit.rb
index d8ad8be..013b6bf 100644
--- a/lib/unitwise/unit.rb
+++ b/lib/unitwise/unit.rb
@@ -88,7 +88,7 @@ module Unitwise
     # @api public
     def scalar(magnitude = 1)
       terms.reduce(1) do |prod, term|
-        prod * term.scalar(magnitude)
+        prod * Number.rationalize(term.scalar(magnitude))
       end
     end
 

It was a problem of a BigDecimal being multiplied with a Rational, looks like, though I won't pretend to understand why that didn't work.

saraid avatar Nov 12 '21 07:11 saraid