Parallel loading with torch.utils.data.DataLoader

[YingtianDt]

Using torch.utils.data.DataLoader, the loading seems to create dead lock. May I ask if this is expected?

[xiao7199]

Are you using cpu(0) with num_of_workers > 1?

[YingtianDt]

I was using the default setting, so yes with cpu(0). It turns out that the problem is solved by initializing the video reader within each worker process, instead of first initializing the readers and send them too the worker process.

[xiao7199]

Could you please share your code for this part? I've been facing a similar issue recently. Thanks!

[YingtianDt]

My code is a part of a larger codebase, but I basically did the following:

from decord import VideoReader
from torch.utils.data import Dataset, DataLoader

class MyDataset(Dataset):
      ...
      def __getitem__(self, index):
            path = self.paths[index]
            frames = VideoReader(path)
            ...

dataset = MyDataset()
dataloader = DataLoader(dataset, ..., num_workers=16)