Open3D icon indicating copy to clipboard operation
Open3D copied to clipboard

Floating-point equality check in RANSAC result comparison may never trigger

Open zzhuolun opened this issue 6 months ago • 0 comments

Checklist

Describe the issue

In PointClooud::SegmentPlane, the RANSAC result comparison includes a direct equality check on fitness_, which is a double:

if (this_result.fitness_ > result.fitness_ ||
    (this_result.fitness_ == result.fitness_ &&
     this_result.inlier_rmse_ < result.inlier_rmse_)) {

Since fitness_ is defined as a double, exact equality is highly unlikely due to precision limitations. This makes the tie-breaking condition based on inlier_rmse_ effectively unreachable in most cases.

Suggestion:

const double epsilon = 1e-8;
if (this_result.fitness_ > result.fitness_ ||
    (std::abs(this_result.fitness_ - result.fitness_) < epsilon &&
     this_result.inlier_rmse_ < result.inlier_rmse_)) {

This would allow more robust tie-breaking when fitness_ values are nearly equal.

zzhuolun avatar May 30 '25 15:05 zzhuolun