triton-inference-server
Error about driver version compatibility
Description
When I tried to using Triton server version 2.51.0(Nvidia Release 24.10) on Orin Nano with Jetpack 6.1, an Error shows:
Triton Information 2.51.0(Nvidia Release 24.10)
Are you using the Triton container or did you build it yourself? just pull the Triton container
To Reproduce
- pull the triton container 24.10 on Jetpack 6.1
- run the triton server with docker
Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).
Expected behavior the Triton server can run, but is there any risk regarding the Error?
Hi @GLW1215, if your driver is incompatible with the build, it's possible that you'll encounter GPU/CUDA related errors at server startup when trying to load a GPU-based model. This should be a quick thing to verify by trying to load a GPU model with Triton.
CC @nv-kmcgill53 for viz
The issue you are facing is a cuda forward compatibility issue. The host machine has the R540 driver installed but the container has the R560 driver installed which seems incompatible from the screenshot you have posted. The easiest fix is to upgrade the host device to the latest R560 driver and then the container and the host have the same installation. If this is not possible then hopefully you can install the cuda compat package on the host machine to allow for driver compatibility. Let me know if this advice doesn't work or isn't applicable to your use case in some way and I can look further into it.
