Warnings when running the streaming flow on GPU

[jingzhaoou]

The offline flow has no warning. However, when I ran the streaming flow, I got the following warnings:

2023-05-25 00:21:23.008301845 [W:onnxruntime:, session_state.cc:1136 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2023-05-25 00:21:23.008326867 [W:onnxruntime:, session_state.cc:1138 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

I turned on ONNX verbose output and found these messages:

2023-05-25 00:21:20.004017871 [V:onnxruntime:, session_state.cc:1116 VerifyEachNodeIsAssignedToAnEp] Node placements
2023-05-25 00:21:20.004031728 [V:onnxruntime:, session_state.cc:1122 VerifyEachNodeIsAssignedToAnEp]  Node(s) placed on [CPUExecutionProvider]. Number of nodes: 875
......
2023-05-25 00:21:20.006623532 [V:onnxruntime:, session_state.cc:1122 VerifyEachNodeIsAssignedToAnEp]  Node(s) placed on [CUDAExecutionProvider]. Number of nodes: 2426
......

So, the warnings are correct as some nodes are assigned to CPU while the others are assigned to GPU. ONNX runtime gave the following explanation:

2023-05-25 00:21:19.287699117 [I:onnxruntime:, fallback_cpu_capability.cc:157 GetCpuPreferredNodes] ORT optimization- Force fallback to CPU execution for node: Unsqueeze_5018 because the CPU execution path is deemed faster than overhead involved with execution on other EPs  capable of executing this node

I wonder if we need to do anything to these warnings and appreciate your feedback. Thanks.

[csukuangfj]

Sorry, we have no idea as of now about how to fix the warnings.