Question: Does ONNX-RT silently fallbacks to CPU?

[Thytu]

trafficstars

Context Hey, I'm setting up a python backend and I am using dlpack to keep the tensors on GPU. As described in its doc, the tensor will be in either CPU or GPU memory, depending on how those tensors were last used, which in my cased is by a ONNX model using an ensemble.

A dummy diagram of the ensemble would be like this:

 ------------                  -------------- 
| ONNX MODEL | ---[Tensor]--> | Python Model |
 ------------                  -------------- 

When doing the request one by one to my ensemble, the Tensor is indeed always on the GPU as expected, but when sending a bunch of request simultaneously to the ensemble, it appears that some times the tensor is instead present on the CPU.

I see only three possible reasons for this behaviour:

1. ONNX-RT/backend does a silent fallback to CPU
2. Triton's queue sometimes move the tensor from GPU to CPU (would be the first time we see this)
3. Another model/service use this tensor (but none of my models does use this same tensor)

Note that both models are set as KIND_GPU, and for debugging purposed the max batch size is set to 1.

Question Is it possible that ONNX-RT or the ONNX Backend does a silent fallback to CPU when certain conditions are met?

[Thytu]

@jsoto-gladia