Pipeline freezes on depthai 2.17.1.0 (development branch)

[f-alemauro]

Hi! This script/pipeline somehow fails, it stops running.

Here is a MWE extracted from my code

import cv2
import depthai as dai
import numpy as np
import os
from os.path import isfile, join


def to_planar(arr: np.ndarray, shape: tuple) -> list:
    return [val for channel in cv2.resize(arr, shape).transpose(2, 0, 1) for y_col in channel for val in y_col]


w_frame = 1920
h_frame = 1080
croptl_x = 0.1
croptl_y = 0.1
cropbr_x = 0.9
cropbr_y = 0.9

cropped_frame_width = round((cropbr_x-croptl_x) * w_frame)
cropped_frame_height = round((cropbr_y-croptl_y) * h_frame)

pipeline = dai.Pipeline()

xlinkIn = pipeline.create(dai.node.XLinkIn)
xlinkIn.setMaxDataSize(w_frame*h_frame*3)
xlinkIn.setStreamName("input")

# creating preview out node
rgbOut = pipeline.createXLinkOut()
rgbOut.setStreamName("rgb")


zoom_manip = pipeline.create(dai.node.ImageManip)
zoom_manip.setCropRect(croptl_x, croptl_y, cropbr_x, cropbr_y)
zoom_manip.setMaxOutputFrameSize(cropped_frame_width * cropped_frame_height * 3)
xlinkIn.out.link(zoom_manip.inputImage)
zoom_manip.out.link(rgbOut.input)

image_manip_script = pipeline.create(dai.node.Script)
zoom_manip.out.link(image_manip_script.inputs['frame'])
with open("script.py", "r") as f:
    image_manip_script.setScript(f.read())

manip_crop = pipeline.create(dai.node.ImageManip)
image_manip_script.outputs['manip_img'].link(manip_crop.inputImage)
image_manip_script.outputs['manip_cfg'].link(manip_crop.inputConfig)
manip_crop.setWaitForConfigInput(True)




list_of_images = [f for f in os.listdir("img") if isfile(join("img", f))]
list_of_images.sort(key=lambda x: int(x.split('.')[0]))

with dai.Device(pipeline) as device:
    info = device.getDeviceInfo()
    inputImage = device.getInputQueue("input")
    qRgb = device.getOutputQueue(name="rgb", maxSize=4, blocking=True)
    
    while list_of_images:
        if list_of_images:
            filename =  list_of_images.pop(0)
            seq_number = int(filename.split('.')[0])
            img_data = cv2.imread(join("img",filename))
            print("=================================")
            print("Sending image {}".format(filename))
            img = dai.ImgFrame()
            img.setData(to_planar(img_data, (w_frame, h_frame)))
            img.setType(dai.RawImgFrame.Type.BGR888p)
            img.setSequenceNum(seq_number)
            img.setWidth(w_frame)
            img.setHeight(h_frame)
            inputImage.send(img)
            inRgb = qRgb.get()
            if inRgb is not None:
                c = inRgb.getCvFrame()
                cv2.imshow('RGB', inRgb.getCvFrame())
        cv2.waitKey(10)

and this is the script

cropxmin = 0.782648096357299
cropymin = 0.06242045680856734
cropxmax = 0.8577302398367053
cropymax =  0.092376426825723
while True:
    node.warn("Face -> ===============")
    try:
        frame = node.io['frame'].get()
        node.warn("Face -> read frame")
        cfg = ImageManipConfig()

        cfg.setCropRect(cropxmin, cropymin, cropxmax, cropymax)
        cfg.setResizeThumbnail(62, 62, 0, 0, 0)
        cfg.setKeepAspectRatio(False) 
        node.io['manip_img'].send(frame)
        node.io['manip_cfg'].send(cfg)
        node.warn("Fce -> manip cfg sent")      
    except Exception as e:
        node.warn(str(e))
        node.warn(e.message)

The pipeline simply:

1. takes a 1920x1080 image in input from the host
2. crop the image
3. send the image to a script/image manip couple. 3.1. The script/manip node crops it again 3.2. The script/manip node resizes it with thumbnail
4. the input image is sent back to the host and displayed

The problem is that for these particolar values of cropping in the script

cropxmin = 0.782648096357299
cropymin = 0.06242045680856734
cropxmax = 0.8577302398367053
cropymax =  0.092376426825723

the pipeline seems to get stuck. It loads the first image and then nothing else. Debug does not help, I get no error indeed.

This is the current config Windows 10 Python 3.9.2 depthai 2.17.1.0.dev0+10b0d5401b4248620687622663627a43b90c3055 (using this develop branch to enable 1080p image loading from host)

Hope you can reproduce this issue. Unfortunately I can't have any further helps for you.

Cheers!

[Luxonis-Brandon]

Thanks for the report. And sorry about the trouble here. We'll try to reproduce internally and debug/fix.

[f-alemauro]

Here is a little update: splitting the script like this

cropxmin = 0.782648096357299
cropymin = 0.06242045680856734
cropxmax = 0.8577302398367053
cropymax =  0.092376426825723
while True:
    try:
        frame = node.io['frame'].get()
        node.warn("Face -> read frame")
        cfg = ImageManipConfig(
        cfg.setCropRect(cropxmin, cropymin, cropxmax, cropymax)
        node.io['manip_img'].send(frame)
        node.io['manip_cfg'].send(cfg)   
    except Exception as e:
        node.warn(str(e))
        node.warn(e.message)

while True:
    try:
        frame = node.io['frame'].get()
        cfg = ImageManipConfig()
        cfg.setResizeThumbnail(62, 62, 0, 0, 0)
        cfg.setKeepAspectRatio(False) 
        node.io['manip_img'].send(frame)
        node.io['manip_cfg'].send(cfg)
    except Exception as e:
        node.warn(str(e))
        node.warn(e.message)

and linking them one after the other make the pipeline work fine, but the cropping seems to be buggy:

regardless the values, cropping in some frame is wrong, it crops "more'' than it should.

These three images have been cropped with very similar values (values coming from a body detection NN), but are quite different, you know what I mean? 😄 And of course the crop is wrong in the second and third image.

This problem does not happen with depthai 3.13.1.0 (the version we are using in prodution, migration to 2.17 in progress 😄

[Luxonis-Brandon]

Thanks for the details. This looks like an error from when we refactored our image manipulation node and we didn't catch this.

We'll dig in.

[f-alemauro]

Thank you @Luxonis-Brandon

Here is a little update: if I add a simple resize like this

cfg.setResize(widht, height)

to the script performing the cropping, the cropping stops being buggy. Here is the final result of the first script:

cropxmin = 0.782648096357299
cropymin = 0.06242045680856734
cropxmax = 0.8577302398367053
cropymax =  0.092376426825723
while True:
    try:
        frame = node.io['frame'].get()
        f_w = frame.getWidth()
        f_h = frame.getHeight()
        cfg = ImageManipConfig()
        cropped_width = round((cropxmax - cropxmin)*f_w)
        cropped_height = round((cropymax - cropymin)*f_h)
        cfg.setCropRect(cropxmin, cropymin, cropxmax, cropymax)
        cfg.setResize(cropped_width, cropped_height)
        cfg.setKeepAspectRatio(False)
        node.io['manip_img'].send(frame)
        node.io['manip_cfg'].send(cfg)   
    except Exception as e:
        node.warn(str(e))
        node.warn(e.message)

cropped_width and cropped_height are calculated to not resize the image actually, I call it a "fake resize". I need to do this as I need to crop the image with letterboxing afterward. Please note alse I set keepAspectRatio to False, otherwise the cropping gets buggy

So, this last script followed by the following one

while True:
    try:
        frame = node.io['frame'].get()
        cfg = ImageManipConfig()
        cfg.setResizeThumbnail(w, h, 0, 0, 0)
        cfg.setKeepAspectRatio(False) 
        node.io['manip_img'].send(frame)
        node.io['manip_cfg'].send(cfg)
    except Exception as e:
        node.warn(str(e))
        node.warn(e.message)

seems to be fine to get my image cropped and resized (I use this to crop out detected bodies and detected faces in the original image). I'll test it throughly.

[Luxonis-Brandon]

Great thanks for circling back!

[Erol444]

@f-alemauro Could you please provide a MRE (in zip) that contains all required files? As there are at least 3 files required (I haven't dug into the 'cycling' of image logic you have used). Thanks, Erik