data icon indicating copy to clipboard operation
data copied to clipboard

Use `DataPipe` not `IterDataPipe` as the type hints for `DataLoader2` related code

Open ejguan opened this issue 2 years ago • 2 comments

🐛 Describe the bug

There are a couple of places that DataLoader2 uses IterDataPipe as the type hint (here, here, etc.)

We should add a type called DataPipe = Union[IterDataPipe, MapDataPipe] as the type hint, and use this new type hint for all functions.

Versions

main branch

ejguan avatar May 23 '22 15:05 ejguan

Let's not label it as the good first issue since the fix might need some work in the internal code base

ejguan avatar May 23 '22 15:05 ejguan

Need to be very careful. As for example here https://github.com/pytorch/data/blob/12cfaf8899b1337981cd4edf9deef127f925f1bd/torchdata/dataloader2/reading_service.py#L17 is should be IterDataPipe as any MapDataPipe supposed to be wrapped to sampler before passing to DataLoader.

VitalyFedyunin avatar Jul 06 '22 18:07 VitalyFedyunin

Closing as it has been solved

ejguan avatar Oct 04 '22 13:10 ejguan