SAM Model Predict Bug

[PtttCode]

I had followed https://github.com/heartexlabs/label-studio-ml-backend/tree/master/label_studio_ml/examples/segment_anything_model to deploy SAM model

But there are some bugs of SAM predictions:

The coordinates of the mask will be offset from the target I expected, but in fact the shape of the mask is correct

Examples:

1. Here is a 2048*1024 image performance(Offset):

2. Here is a 640*426 image performance(Correct):

3. Here is a 640*480 image performance(Offset):

There are absolutely bugs on the process after prediction, and I think the key of question is the image padding

Here are the original images:

[jflessau]

Probably related to this issue with the ONNX model: https://github.com/anuragxel/salt/issues/2

Seems like masks are off for input images not matching the aspect ratio (2:3) set with

"orig_im_size": torch.tensor([1500, 2250], dtype=torch.float),

in onnxconverter.py

Your example image, with a size of 640x426, happens to have the correct aspect ratio.

[shondle]

Hi, this issue was addressed in the pull request mentioned. @jflessau thank you so much for pointing me in the right direction. This new pull request (if using the AdvancedSAM backend instead) should have no issues with masks offsetting from images -> this is an ONNX issue described in the README of this pull request.