geo icon indicating copy to clipboard operation
geo copied to clipboard

ppc64le - Fix deviation in floating point values caused by FMA

Open prashantkhoje opened this issue 2 years ago • 6 comments

This was exposed while testing CockroachDB. Following tests from pkg/sql/opt/exec/execbuilder fail due to floating point precision differences caused by FMA:

  • TestExecBuild/local/geospatial
  • TestExecBuild/local-vec-off/geospatial
  • TestExecBuild/local-spec-planning/geospatial
  • TestExecBuild/fakedist/geospatial
  • TestExecBuild/fakedist-vec-off/geospatial
  • TestExecBuild/fakedist-metadata/geospatial
  • TestExecBuild/fakedist-disk/geospatial
  • TestExecBuild/fakedist-spec-planning/geospatial

With explicit casts in this patch, these failures are resolved.

References: https://go.dev/ref/spec#Floating_point_operators https://github.com/golang/go/issues/53297 https://github.com/golang/go/issues/48145 https://github.com/disintegration/gift/issues/20#issuecomment-524551797

prashantkhoje avatar Jul 27 '22 12:07 prashantkhoje

@rsned, please review.

prashantkhoje avatar Jul 27 '22 12:07 prashantkhoje

@otan, fyi

prashantkhoje avatar Aug 18 '22 04:08 prashantkhoje

@ambujkumar3 fyi

prashantkhoje avatar Aug 26 '22 05:08 prashantkhoje

Can you see if this still occurs with the latest updates.

rsned avatar Apr 21 '23 00:04 rsned

Yes. Cockroachdb still encounters FMA failures with the latest golang/geo. The changes proposed here fixes the deviation.

prashantkhoje avatar May 27 '23 16:05 prashantkhoje