problem with nvidia.dali.plugin.paddle.DALIGenericIterator

[thgpddl]

When I use DALIGenericIterator in nvidia.dali.plugin.paddle to build fast data loading code, some problems occur. The details are as follows:

from nvidia.dali import pipeline_def
import nvidia.dali.fn as fn
from nvidia.dali.plugin.paddle import DALIGenericIterator

# create pipeline
@pipeline_def
def file_reader_pipeline():
    jpegs, labels = fn.readers.file(
                                    file_root="../dataset/train",
                                    random_shuffle=True,
                                    name='Reader')
    images = fn.decoders.image(jpegs, device='mixed')

    return images, labels


# create DALIGenericIterator
pipes = [file_reader_pipeline(batch_size=128, num_threads=2)]
dali_iter = DALIGenericIterator(pipelines=pipes,
                                output_map=['data', 'label'],
                                reader_name='Reader')

Traceback (most recent call last):
  File "/home/ubuntu/WorkSpace/micronet-ppv1/data/dali_doc.py", line 21, in <module>
    reader_name='Reader')
  File "/home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/plugin/paddle.py", line 260, in __init__
    self._first_batch = DALIGenericIterator.__next__(self)
  File "/home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/plugin/paddle.py", line 277, in __next__
    outputs = self._get_outputs()
  File "/home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/plugin/base_iterator.py", line 297, in _get_outputs
    outputs.append(p.share_outputs())
  File "/home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/pipeline.py", line 997, in share_outputs
    return self._pipe.ShareOutputs()
RuntimeError: Critical error in pipeline:
Exception in CPU stage: [/opt/dali/dali/pipeline/executor/executor.cc:72] Assert on "device_id_ == CPU_ONLY_DEVICE_ID" failed: Wrong device_id provided, it should be >= 0, or equal to CPU_ONLY_DEVICE_ID.
Stacktrace (8 entries):
[frame 0]: /home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/libdali.so(+0x8199f) [0x7f89d238399f]
[frame 1]: /home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/libdali.so(dali::Executor<dali::AOT_WS_Policy<dali::UniformQueuePolicy>, dali::UniformQueuePolicy>::RunCPUImpl()+0x8fa) [0x7f89d24002ea]
[frame 2]: /home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/libdali.so(dali::Executor<dali::AOT_WS_Policy<dali::UniformQueuePolicy>, dali::UniformQueuePolicy>::RunCPU()+0xe) [0x7f89d24006ce]
[frame 3]: /home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/libdali.so(+0xb503d) [0x7f89d23b703d]
[frame 4]: /home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/libdali.so(+0x12dc44) [0x7f89d242fc44]
[frame 5]: /home/ubuntu/miniconda3/envs/pp/lib/python3.7/site-packages/nvidia/dali/libdali.so(+0x70cf4f) [0x7f89d2a0ef4f]
[frame 6]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f89d3fda609]
[frame 7]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f89d3eff133]

Current pipeline object is no longer valid.

Process finished with exit code 1

[jantonguirao]

You are using a mixed (using GPU) backend operator (fn.decoders.image) but not providing a valid device_id for the pipeline. I think the error comes from there. Can you please try setting the device id to the pipeline? file_reader_pipeline(batch_size=128, device_id=0, num_threads=2) ?

[jantonguirao]

Closing this issue now. If you still need help, please reopen.