"Payload data has been discarded" when using tensorflow lite in a separate thread.

[robertatdm]

Describe what you want to implement and what the issue & the steps to reproduce it are:

This is hard to reproduce because I used a custom embedded board with an i.MX8M+ and a custom Linux (v6.1.24). A little advice for further debugging would be helpful.

I am trying to run a python application with two threads. The camera thread continuously grabs new images and prints out the mean value of the image pixels (just for testing).

class CameraThread(threading.Thread):
    def __init__(self):
        super().__init__()
        self.camera = None
        self.frame = None
        self.w_aq = 2160    # Width of aquisited image from camera.
        self.h_aq = 1620
        self.running = False
        
    def connect_camera(self):
        self.camera = pylon.InstantCamera(pylon.TlFactory.GetInstance().CreateFirstDevice())
        if not self.camera:
            print("Could not connect to camera")

    def set_camera_parameters(self):
        self.camera.Open()
        ...
        self.camera.AcquisitionFrameRateEnable.SetValue(True)
        self.camera.AcquisitionFrameRate.SetValue(20.0)
        self.camera.DeviceLinkThroughputLimitMode.SetValue("On")
        self.camera.DeviceLinkThroughputLimit.SetValue(200000000)
        ...

    def run(self):
        self.running = True
        self.connect_camera()
        self.set_camera_parameters()
        
        self.frame = np.full((self.camera.Height.Value, self.camera.Width.Value, 3), 123, np.uint8)
        
        self.camera.StartGrabbing(pylon.GrabStrategy_LatestImageOnly)
        while self.running:
            try:
                res = self.camera.RetrieveResult(5000, pylon.TimeoutHandling_Return)
                if(res.GrabSucceeded()):
                    im = res.GetArray().copy()
                    print(int(im.mean()), end=' ')
                else:
                    print("WARNING: Grab didn't succeed")
                    print(res.GetErrorDescription())
                res.Release()
            except Exception as e:
                print(e)
                self.running = False

        if self.camera:
            self.camera.StopGrabbing()
            self.camera.Close()

    def stop(self):
        self.running = False

Then there is another thread which repeatedly runs a tensorflow lite model:

class InferenceThread(threading.Thread):
    def __init__(self):
        super().__init__()      

        self.running = False

        self.interpreter = tfl.Interpreter(
            "retinanet/model_int8.tflite",  
            experimental_delegates=[tfl.load_delegate("/usr/lib/libvx_delegate.so")]  
        )

        self.input_details = self.interpreter.get_input_details()   
        self.output_details = self.interpreter.get_output_details()
        self.interpreter.allocate_tensors()
        self.images_shape = self.input_details[0]['shape']

        # First time invoke (Takes much time)
        dummy_data = np.random.uniform(0.0, 255.0, self.images_shape).astype('uint8')
        self.interpreter.set_tensor(self.input_details[0]['index'], dummy_data.copy())
        self.interpreter.invoke()

    def process_image(self):
        dummy_data = np.random.uniform(0.0, 255.0, self.images_shape).astype('uint8')
        self.interpreter.set_tensor(self.input_details[0]['index'], dummy_data.copy())
        self.interpreter.invoke()
        
    def run(self):
        self.running = True
        while self.running:
            time.sleep(0.8)
            self.process_image()
    
    def stop(self):
        self.running = False

Here is the rest of the code for completeness:

import signal, threading, time
import numpy as np
from pypylon import pylon
import tflite_runtime.interpreter as tfl

if __name__ == "__main__":
    signal.signal(signal.SIGINT, signal.SIG_DFL)
    camera_thread = CameraThread()
    inference_thread = InferenceThread()

    camera_thread.start()
    inference_thread.start()
    
    camera_thread.join()
    inference_thread.join()

The grabbing works for a few images, but occasionally I get an unsuccessful GrabResult with this ErrorDescription: Payload data has been discarded. Payload data can be discarded by the camera device if the available bandwidth is insufficient. I can use other computationally expensive threads, but the issue arises only when I use tensorflow lite with libvx_delegate, that uses the built-in NPU of the i.MX8M+. I have no Idea, how those two unrelated libraries can interfer with each other.

The test works fine on other systems (NXP i.MX8M+ evaluation board / Ubuntu Host-PC).

It would be great to know more about this error and what can potentially cause it. Thanks in advance.

Is your camera operational in Basler pylon viewer on your platform

Yes

Hardware setup & camera model(s) used

Camera: Basler dart da3840-45uc

• Connected via a single USB-micro-B cable

System: Unfortunately this is all very customized and can have lot's of error causes.

• A custom Single Board Computer with an i.MX8MPlus and USB-3.0 interface for the camera.
• A custom Board Support Package merged with the meta-freescale and meta-imx yocto layers.
  • Linux kernel (v6.1.24)

Runtime information:

python: 3.10.9 (main, Dec  6 2022, 18:44:57) [GCC 11.3.0]
platform: linux/aarch64/6.1.24
pypylon: 1.9.0+pylon6.2.0 / 6.2.0.18677