data
data copied to clipboard
[Nodes] Ensure get_worker_info works correctly in ParallelMapper
🚀 The feature
Support for torch.utils.data.IterableDataset
Motivation, pitch
Currently IterableDataset (and possibly some map datasets) rely on torch.utils.data.get_worker_info. To ensure drop-in compatibility, we should make sure this works correctly for ParallelMapper by setting the environment up correctly. We could do this directly in ParallelMapper, or in a new IterableDatasetWrapper class.
Alternatives
No response
Additional context
No response