Spatial detection giving wrong results when using letterboxing method to increase FOV

[medonni]

Hi!

For some time now i've been working on incorporating the Oak-D Lite camera into my app and everything went pretty normal untill yesterday. I've previously sucessfully added the spatial detection on my pre-trained YOLOv5 model and it worked pretty accurately, with the margin of error of 1-2cm. The issue rose when i added letterboxing to increase the FOV of the camera, as when using the on-device decoding i was stuck to the squared screen and the FOV was not that great. Now the spatial calculator is pretty off, giving results such as 12meters when it reality is more like 50cm. The bounding box on the depth heatmap seems to be stuck in the top left corner of the screen, and when moving the object closer to that area the results get normal. Moving the object anywhere else aside from the corner gives insanely accurate results.

This would be the minimum reproductive code.

https://pastebin.com/wBqyfYKm

[madgrizzle]

I had this exact issue and there was a thread created in the #depth discord channel called "letterbox mode + spatial detection" and it addressed this. Im travelling and can't figure out how to link to it from my phone.

[Erol444]

@madgrizzle Found this discussion on Discord here.

[medonni]

So what i get from this, is that i should change the camRgb.setIspScale(1, 5) parameter to camRgb.setIspScale(2, 3) ? Doing that openCV throws me the following error:

OpenCV(4.5.5) d:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\color.simd_helpers.hpp:108: error: (-215:Assertion failed) sz.width % 2 == 0 && sz.height % 3 == 0 in function 'cv::impl::anonymous-namespace'::CvtHelper<struct cv::impl::anonymous namespace'::Set<1,-1,-1>,struct cv::impl::A0x6b3065a4::Set<3,4,-1>,struct cv::impl::A0x6b3065a4::Set<0,-1,-1>,1>::CvtHelper'

I assume i'm supposed to change something else too, but im not sure what exactly.

[SzabolcsGergely]

@medonni the suggested way was using image "squashing" to preserve FOV, like here: https://github.com/luxonis/depthai-python/blob/main/examples/MobileNet/rgb_mobilenet_4k.py#L41 Not immediately sure what is the issue in your case using ImageManip, but will check out.

[medonni]

@medonni the suggested way was using image "squashing" to preserve FOV, like here: https://github.com/luxonis/depthai-python/blob/main/examples/MobileNet/rgb_mobilenet_4k.py#L41 Not immediately sure what is the issue in your case using ImageManip, but will check out.

Thanks, meanwhile I will check the "squashing" method you provided.