data
data copied to clipboard
pytorch
A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.
### 🚀 The feature, motivation and pitch When training with a dataloader, we might stop training in the middle of the run, and start it again later. Then, we usually...
### 🚀 The feature How does one create an `IterDataPipe` with [`scan`/`fold`](http://learnyouahaskell.com/higher-order-functions) semantics? ### Motivation, pitch Necessary for pipelines that require some kind of state, eg. label encoding for an...
### 🐛 Describe the bug 1. Create a pipeline and use MPRS with 'spawn' (same behavior might potentially also be replicated with 'fork') 2. Have the worker process fail to...
### 🚀 The feature Add the ability to automatically invalidate a cached sub-graph when the remote files change after being cached locally. ### Motivation, pitch Say I have multiple files...
### 📚 The doc issue Similar to #1104, `DataLoader2` has built-in functionality to set random seed for DataPipes when they have `set_seed` method implemented. https://github.com/pytorch/data/blob/main/torchdata/dataloader2/graph/settings.py#L22 We should provide official guidance...
### 📚 The doc issue TorchData has a few special graph functions to handle Shuffle and Sharding DataPipe. But, we never document what is expected for those graph functions, which...
### 🚀 The feature ``` import torchdata.datapipes as dp from torch.utils.data.datapipes.iter.sharding import SHARDING_PRIORITIES from torchdata.dataloader2 import MultiProcessingReadingService, DataLoader2 my_worker_info = None def abc(x): return x * my_worker_info.worker_id def worker_init(dp, worker_info):...
### 🚀 The feature Sorry if this is already possible, but I didn't found anything about this in the documentation. Can the same MPRS, i.e. also the same set of...
### Changes - Update LAION-example to use threadpool_map - Change default value of scheduled_tasks to 500
Similar to #901 and as discussed in #850 this migrates examples in `iter/util` directory to sphinx doctest format. As sphinx is also able to check normal doctest-style examples (which will...