Running on GPU - Prediction not happening

[Avinash0795]

Respected Sir,

Sorry to post with so many Questions. I tried to run the Application on GPU. It is way faster than Running on CPU but when it is running on GPU, there seems to be no prediction happening and therefore the Text is not being rendered. It would be really great if you could shed some light upon it. As per the documentation I understand that the session is being created on the GPU for Evaluation, but should I also load the model in GPU? But upon running on GPU, the Predictions are not happening. It would be great to have a reply

[doughtmw]

Hi @Avinash0795, the GPU support for ONNX model inference on the HL2 is a bit strange—I haven't had much success with getting it to work correctly or achieving better performance than CPU inference. There are some additional details on another repo of mine (here's a link), but in short, you may need to play around with using the default Windows.AI.MachineLearning library or including an additional NuGet package to use the Microsoft.AI.MachineLearning library (here's a link with some information) which has better support for a range of ONNX model opsets and CPU/GPU inference.