[QUESTION] How to run YOLOX on Oak?

[domef]

I'm trying to deploy YOLOX on an Oak device. I can load the model but I don't know how extract the output of the model.

I downloaded the YOLOX Nano ONNX from the official repo and I converted the .onnx to .blob using blobconverter.

This is my code:

import depthai as dai
import time
import numpy as np
import cv2

pipeline = dai.Pipeline()
pipeline.setOpenVINOVersion(version=dai.OpenVINO.Version.VERSION_2021_4)

# camera
center_camera = pipeline.createColorCamera()
center_camera.setPreviewSize(256, 256)
center_camera.setBoardSocket(dai.CameraBoardSocket.RGB)
center_camera.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
center_camera.setInterleaved(False)
center_camera.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)

# neural network
nn = pipeline.createNeuralNetwork()
nn.setBlobPath("/path/to/blob")

# streams
center_stream = pipeline.createXLinkOut()
center_stream.setStreamName("rgb")
center_camera.preview.link(center_stream.input)
center_camera.preview.link(nn.input)
nn_stream = pipeline.createXLinkOut()
nn_stream.setStreamName("nn")
nn.out.link(nn_stream.input)

with dai.Device(pipeline.getOpenVINOVersion()) as device:
    device.startPipeline(pipeline)

    q_rgb = device.getOutputQueue("rgb")
    q_nn = device.getOutputQueue("nn")

    while True:
        in_rgb = q_rgb.tryGet()
        in_nn = q_nn.tryGet()

        if in_rgb is not None and in_nn is not None:
            print('names', in_nn.getAllLayerNames())
            x = in_nn.getData()
            img_rgb = in_rgb.getFrame().transpose(1, 2, 0)
            out = in_nn.getFirstLayerFp16()
            out = np.array(out).astype(np.uint8).reshape(256, 256)
            cv2.imshow('img', cv2.cvtColor(img_rgb, cv2.COLOR_RGB2BGR))
            cv2.imshow('out', out)
            cv2.waitKey(1)

        time.sleep(0.01)

It runs, so I think the blob has been loaded correctly but I don't understand how the get the detections.

I visualized the YOLOX onnx model with netron, the output node is named output, whereas the result of getAllLayerNames() is '1'. Also with getFirstLayerFp16() I get the input in grayscale.

I also tried the YoloDetectionNetwork node without success.

How can I deploy YOLOX on Oak device?

[VanDavv]

Hi @domef, we had an initial success with YoloX and released an experiment here. However, as mentioned in a PR for this experiment, the blob that is being used is not the latest and was not yet optimized to apply mean/std with model optimizer, and had to be applied manually. We'll definitely circle back to this, but in the meantime it can serve as a starting point for anyone who wants to try yolox or try to update to latest blob