DALI icon indicating copy to clipboard operation
DALI copied to clipboard

decoders.video CUDA_ERROR_SHARED_OBJECT_SYMBOL_NOT_FOUND (302):

Open Voveka98 opened this issue 3 weeks ago • 3 comments

Describe the question.

Hi! I have nvidia-dali pipeline where i try to decode video on GPU. I have nvidia-dali-cuda120==1.51.2

My pipeline looks next way

import nvidia.dali.fn as fn
import nvidia.dali.types as types
from nvidia.dali.plugin.triton import autoserialize
import os 
from nvidia.dali import pipeline_def
OUTPUT_DIR = os.path.dirname(os.path.abspath(__file__))


@pipeline_def(batch_size=1, num_threads=2, device_id=0)
def video_decode_pipeline():
    video_data = fn.external_source(name="DALI_INPUT_0", device="cpu", dtype=types.UINT8)
    frames = fn.experimental.decoders.video(video_data, device="mixed")
    return frames



if __name__ == "__main__":
    pipe = video_decode_pipeline(device_id=0)
    output_fn = os.path.join(OUTPUT_DIR, "model.dali")
    pipe.serialize(filename=output_fn)

With config.pbtxt

name: "keyframes_dali"
backend: "dali"
max_batch_size: 1

input [
  {
    name: "DALI_INPUT_0"
    data_type: TYPE_UINT8
    dims: [ -1 ]
  }
]

output [
  {
    name: "DALI_OUTPUT_0"
    data_type: TYPE_UINT8
    dims: [ -1, -1, 3 ]
  }
]

instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
] 

It successfully loaded to Triton Inference Server but when i try to infer model with bytes of video file i get next error

InferenceServerException: [StatusCode.UNKNOWN] Runtime error: Critical error in pipeline:
Error when executing Mixed operator experimental__decoders__Video encountered:
Error in thread 0: CUDA driver API error CUDA_ERROR_SHARED_OBJECT_SYMBOL_NOT_FOUND (302):
shared object symbol not found
Current pipeline object is no longer valid.

I tried to look for this error but had no success in it. Can you help me what should i check?

Check for duplicates

  • [x] I have searched the open bugs/issues and have found no duplicates for this bug report

Voveka98 avatar Dec 09 '25 12:12 Voveka98

Hi @Voveka98,

Thank you for reaching out. It appears there may be a mismatch between your CUDA installation and the DALI version.

Could you please share a minimal, complete set of reproduction steps that we can run on our side? You can use sample data from DALI_extra if needed.

JanuszL avatar Dec 09 '25 14:12 JanuszL

Hi @JanuszL! Thanks for answering Inside my Triton Inference Server docker i have next output from nvidia-smi

triton@jupyter-user:~$ nvidia-smi
Wed Dec 10 05:54:57 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          Off |   00000000:9B:00.0 Off |                    0 |
| N/A   34C    P0            120W /  700W |     591MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

For nvidia dali i got wheel from this index https://pypi.nvidia.com/nvidia-dali-cuda120/ with version nvidia_dali_cuda120-1.51.2-py3-none-manylinux2014_x86_64.wh

Also in TIS docker i dont have ffmpeg, is it required?

Voveka98 avatar Dec 10 '25 05:12 Voveka98

Hi @Voveka98,

Thank you for your response.

Unfortunately, the steps you've provided don’t give me enough information to reproduce the DALI error on my end.
Regarding your note:

For nvidia dali I got the wheel from this index https://pypi.nvidia.com/nvidia-dali-cuda120/ with version nvidia_dali_cuda120-1.51.2-py3-none-manylinux2014_x86_64.whl

Please note that the DALI Triton backend inside the container is compiled for a specific DALI version, so using a different wheel may cause compatibility issues.

Could you please provide a detailed, step-by-step guide that I can follow to reproduce the error? This will help us investigate the issue more effectively.

JanuszL avatar Dec 10 '25 06:12 JanuszL