data Validation of `fn` on `input

Validation of `fn` on `input_col` for Datapipes

Open bushshrub opened this issue 2 years ago • 10 comments

Fixes #362

Document input_col behavior on FlatMapperIterDataPipe
Checks the arguments of mapping function against input_col in FlatMapperIterDataPipe

Jun 10 '22 12:06 bushshrub

Oh, hold on. This will break test_flatmap_iterdatapipe

Jun 10 '22 12:06 bushshrub

This breaks test_map_batches_iterdatapipe, specifically 683 (since there is only 1 argument)

Jun 10 '22 13:06 bushshrub

@ejguan, should I modify fn_2_cols or is that not desired?

Jun 10 '22 13:06 bushshrub

Could you please open a PR in PyTorch core by creating the validation function first?

Mapper
Filter Then, TorchData can import such validation function from PyTorch Core and reuse it.

Jun 10 '22 13:06 ejguan

Oh, good idea. Any specific file I should put it in?

I was thinking torch/utils/data/datapipes/datapipe.py or maybe torch/utils/data/datapipes/utils/__init__.py

Jun 10 '22 13:06 bushshrub

You can put it in https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/utils/common.py

Jun 10 '22 14:06 ejguan

Are the CI failures normal?

Jun 11 '22 01:06 bushshrub

If #293 goes through this function may need to be copied back over here.

Right now I am waiting for core to merge my changes

Jun 15 '22 05:06 bushshrub

@ejguan Okay to continue working on this now that core has merged?

Jun 24 '22 05:06 bushshrub

@ejguan Okay to continue working on this now that core has merged?

Yeah. You can do that. But, you might wait until my PR landed as your PR breaks a few tests for TorchVision and I fixed them in my PR.

Jun 24 '22 14:06 ejguan

Let me close and reopen this now that the validator function has been merged.

Aug 25 '22 01:08 bushshrub