RKNNLite API Inferencing Slow Compared to C++

[Eliasin]

Hi,

I noticed that during my testing, inferencing YOLOv8n with the rknnlite API seems to be much slower than using the C API for inferencing. I measured the time taken like this

        start_time = time.time()
        outputs = self.rknn_lite.inference(inputs=[frame])
        end_time = time.time()
        print(f"Inference duration: {end_time - start_time}")

and found that the times were around ~300 ms when the C version has inference times of around 20 ms on the Rk3588. I am using the 1.6.0 version of librknnrt and an unmodified, converted YOLOv8n model from rknn_model_zoo.

[shubhambfw]

hii @Eliasin , could you please share the details of C api any reference code or document. I want to use it for ResNet inference.

[Eliasin]

hii @Eliasin , could you please share the details of C api any reference code or document. I want to use it for ResNet inference.

The C api reference consists of a pdf, here is a link.

[jdchmiel]

fyi time.time() gives weird results which is why time.perf_counter exists for timing code executions

[Leo5050xvjf]

hii @Eliasin , could you please share the details of C api any reference code or document. I want to use it for ResNet inference.

The C api reference consists of a pdf, here is a link.

Hello, have you solved this issue? I've encountered something similar on the RK3576. I haven't yet compared the inference time differences between the C++ and Python APIs. However, I've noticed that the inference time on the Python board is almost three times slower compared to connected board inference. The FPS in connected board inference is similar to what's listed in the rknn model zoo, but direct Python inference on the board is three times slower. I've already ruled out factors such as model pre-processing, post-processing, and frame reading.