Issue Reproducing Results with eval_3dovs.ipynb

[Alan0817]

Thank you for your great work! I am attempting to reproduce the results by following the commands in the README and using the newly uploaded eval_3dovs.ipynb for evaluation. However, I encountered some version conflicts between the newly uploaded code and the original implementation, such as the get_point_features property in the feature Gaussian model.

I manually addressed these issues, but unfortunately, my results do not match those reported in the paper. Could you kindly clarify if I might have missed any details or if there is an alternative training recipe I should follow?

If you plan to update the code to resolve these inconsistencies, I would truly appreciate it.

[Jumpat]

Hi, I wonder the gap between your reproduced results with our provided results. This may happen due to some wrongly set hyper parameters. It is normal if the gap is acceptable, since there is some randomness in both the training and eval phases.

[Alan0817]

Thank you for your reply. I tested on the 3DOVS-sofa scene and got approximately 70% mIoU (I provided visualization below). My hyperparameter settings follow the provided arguments and command lines. Could you share your hyperparameters?

[Jumpat]

I guess this may caused by the SAM preprocessing phase: the target may be over segmented or wrongly segmented with other objects. You can try these hyperparameters for SAM extractor:

  mask_generator = SamAutomaticMaskGenerator(
      model=sam,
      points_per_side=16,
      pred_iou_thresh=0.6,
      box_nms_thresh=0.7,
      stability_score_thresh=0.1,
      crop_n_layers=0,
      crop_n_points_downscale_factor=1,
      min_mask_region_area=200,
  )