Kevin Tse

Results 85 comments of Kevin Tse

The argument `input_col` should allow you to do that with [`map`](https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Mapper.html?highlight=mapper#torchdata.datapipes.iter.Mapper). You need to download the latest version though. ```python dp = IterableWrapper([("a", 1), ("b", 2)]).map(fn=lambda char: char + char,...

Adding `map_tuple` and `map_dict` should be easy. It will mostly just be wrappers around the existing `map` implementation (i.e. passing `key` of dict to `input_col` and `idx` of tuple to...

@msaroufim I think the API/functionality discussed here is different from `itertools`, but nonetheless I see your point. I just opened #756 to discuss.

Potential duplicate issues of: * #649 * #655 * #661 We should keep one and close the other ones?

> @NicolasHug is there a setup script I need to run beforehand? https://github.com/pytorch/data/runs/7868534954?check_suite_focus=true#step:9:23 That only works on the internal AI cluster. I will message you more details.

> I personally think so as fast forward would be the backup way to run snapshot. It would be better to use __getstate__ to consolidate the state of DataPipe and...

> If you call `dl.shutdown()` at the end, is the problem still persistent? I can confirm that it does fix it. I think the issue is that `reading_service.finalize()` never get...

@nivekt has imported this pull request. If you are a Facebook employee, you can view this diff [on Phabricator](https://www.internalfb.com/diff/D38868731).

> One more though, we deepcopy RS, but we pickle-unpickle DP. I think we need to pick consistent approach. Can you elaborate on this? What make a consistent approach matter...