data icon indicating copy to clipboard operation
data copied to clipboard

[Nodes] Ensure get_worker_info works correctly in ParallelMapper

Open andrewkho opened this issue 1 year ago • 0 comments

🚀 The feature

Support for torch.utils.data.IterableDataset

Motivation, pitch

Currently IterableDataset (and possibly some map datasets) rely on torch.utils.data.get_worker_info. To ensure drop-in compatibility, we should make sure this works correctly for ParallelMapper by setting the environment up correctly. We could do this directly in ParallelMapper, or in a new IterableDatasetWrapper class.

Alternatives

No response

Additional context

No response

andrewkho avatar Dec 09 '24 17:12 andrewkho