Centered instance model appears to apply crop after scaling (rather than before)

[olinesn]

Bug description

When training a centered instance model and scaling the input, the visualization of validation during training appears to show a much wider crop than expected. My guess is that the crop is being applied after scaling, rather than before.

Expected behaviour

Visualization of the centered instance model's validation during training would show a down-sampled (scaled) image that is a crop, or subset, of the larger video.

Actual behaviour

Visualization shows a crop that is much larger than the video itself.

Your personal set up

• OS: Win11

• Version(s): 1.5.2

• SLEAP installation method (listed here): powershell install

Environment packages

# paste output of `pip freeze` or `conda list` here

Logs

# paste relevant logs here, if any

Screenshots

The above is sourced from a video that's 1200h x 1680 w, with a scale of 0.5 and a crop of 832 pixels.

[gitttt-1234]

Hi @olinesn !

If you applied input_scaling to a centered-instance model (say 0.5) and provide a crop size manually, then we do resize the image (0.5) and then crop for the given crop size (in your case 832) . If you want to resize the crop (crop-size = 416), then I would recommend either scaling the crop size too when entering manually (set to 416), or set crop size to "auto" in the GUI, to reduce the crop size to 416px when setting input scaling to 0.5.

Let us know if you have any questions!

Thanks,

Divya

[talmo]

Hey @gitttt-1234, I think the more intuitive behavior should be to specify the crop size in the original scale. This reduces coupling between the parameters and makes it easier to specify manually.

I know it makes the padding logic a bit trickier but I think it makes more sense to do everything with respect to the original resolution.

A precedent for this design is the output stride, which is the same regardless of input scale or max stride.

[olinesn]

Could we have a bounding box on the image preview in the centered instance GUI that represents the crop visually? That way it's super intuitive. In addition to the blue bounding box for the receptive field, there could be a red, dotted line bounding box for the crop.

[gitttt-1234]

Hi @olinesn !

This is updated in sleap-nn to apply scaling to the cropped image. Now, if crop size is set to 800 and scaling is 0.5, the input image to the model would effectively be 400 x 400. We also added a red box displaying the crop size in the GUI alongside the receptive field in #2483.

Thanks,

Divya