geo
geo copied to clipboard
ppc64le - Fix deviation in floating point values caused by FMA
This was exposed while testing CockroachDB. Following tests from pkg/sql/opt/exec/execbuilder fail due to floating point precision differences caused by FMA:
- TestExecBuild/local/geospatial
- TestExecBuild/local-vec-off/geospatial
- TestExecBuild/local-spec-planning/geospatial
- TestExecBuild/fakedist/geospatial
- TestExecBuild/fakedist-vec-off/geospatial
- TestExecBuild/fakedist-metadata/geospatial
- TestExecBuild/fakedist-disk/geospatial
- TestExecBuild/fakedist-spec-planning/geospatial
With explicit casts in this patch, these failures are resolved.
References: https://go.dev/ref/spec#Floating_point_operators https://github.com/golang/go/issues/53297 https://github.com/golang/go/issues/48145 https://github.com/disintegration/gift/issues/20#issuecomment-524551797
@rsned, please review.
@otan, fyi
@ambujkumar3 fyi
Can you see if this still occurs with the latest updates.
Yes. Cockroachdb still encounters FMA failures with the latest golang/geo. The changes proposed here fixes the deviation.