The warning from `onnxruntime::VerifyEachNodeIsAssignedToAnEp`

[Chris-fullerton]

I got this when I try to run insightface on Windows10 with CUDA/cuDNN:

2023-02-18 00:41:44.1541586 [W:onnxruntime:, session_state.cc:1136 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-02-18 00:41:44.1608231 [W:onnxruntime:, session_state.cc:1138 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

The warning msg came from here. I know it seems not like the business of insightface, but according to the comments above the codes about generating warning msg:

If the user explicitly included the CPU provider anyway, then remain silent, but if it was implicitly added, and unexpected fallback happened to a non-preferred provider, warn the user.

Seems that if I set ['CUDAExecutionProvider', 'CPUExecutionProvider'] as provider arg of insightface.app.FaceAnalysis(), this warning msg should not be raised, but it would be raised whether I remove 'CPUExecutionProvider' from providers or not.

Any checking tips about this issue?

[nttstar]

It tells you that some operators will be assigned to be run on CPU, so you should specify the CPU provider in the init option.

[Chris-fullerton]

It tells you that some operators will be assigned to be run on CPU, so you should specify the CPU provider in the init option.

Thanks for your answer, it solves my problem.

[ysj-xuanyuan]

I also reported this warning when using onnxruntime to infer the quantized model of int8, and the model did not infer on the GPU as expected, but instead on the CPU. May I ask what is the reason for this?

[AlyaNovikova]

I also reported this warning when using onnxruntime to infer the quantized model of int8, and the model did not infer on the GPU as expected, but instead on the CPU. May I ask what is the reason for this?

Hi, I also have the same error. Did you happen to figure out how to solve it?

[1145141919]

I also reported this warning when using onnxruntime to infer the quantized model of int8, and the model did not infer on the GPU as expected, but instead on the CPU. May I ask what is the reason for this?

I get the same error.Do you kmow how to duel with it?

[surajrao2003]

Hey there!! So I have been facing a similar kind of issue when I am trying to inference using a dynamically quantized yolov8s onnx model. I have used yolov8s.pt and exported it to yolov8.onnx using onnx export. Then I quantized the onnx model using dynamic quantization (uint8) method provided by onnxruntime which reduced the model size by around 4 times. Though the quantized model worked fine while inferencing on CPU (CPUExecutionProvider), it gives low fps (frames per sec) while inferencing on GPU using CUDAExecutionProvider.

Mentioning both providers CUDAExecutionProvider and CPUExecutionProvider does make the warnings disappear but what I am concerned is about why some nodes are forcefully being executed on the CPU and not CUDA? Is it because CUDA does not yet provide support for such quantized nodes or is there any other particular reason for it?

When checked using log.severity, got this information

import onnxruntime as ort ort.set_default_logger_severity(1)

I am curious about why CUDA is not able to handle these nodes? Because in my opinion it is very unusual to see GPU inference giving a lower fps as compared to CPU inference. Is there any solution to run all the nodes exclusively on CUDA?

[Dryhten-tech]

providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] This line of code allows you to stop displaying warning messages.