data

data copied to clipboard

Loading `.tfrecords` files that require a deserialization method

1

### 🐛 Describe the bug Hi, I have a dataset in TFRecords format and am trying to move to TorchData's API for loading tfrecords files. This is the minimal example:...

fteufel

Passing dict in datapipe/dataset will have memory leak problem

4

### 🐛 Describe the bug Passing dict in datapipe or dataset will casuse memory leak ```python from copy import deepcopy import gc from memory_profiler import profile import torch from torch.utils.data...

qmpzzpmq

Crash when resetting datapipe with bufffer of filehandles

3

### 🐛 Describe the bug When the datapipe iterator is reset, the multiprocessing reading service tries to pickle the datapipe (why?). In case the data pipe contains a buffer with...

falckt

`header()` causes DataLoader with MPRS to hang

3

### 🐛 Describe the bug Using `header()` on an `IterDataPipe` causes `DataLoader` with `MPRS` to hang on the second time thru. ```python from torchdata.dataloader2 import DataLoader2 from torchdata.dataloader2 import MultiProcessingReadingService...

wchang-apixio

Calling __iter__ twice on DataLoader2 causes hang with MPRS

2

### 🐛 Describe the bug I'm aware torchdata isn't being maintained anymore, but thought I'd post this here for posterity: When using `iter` twice for the same instance of DataLoader2,...

JohnHBrock

Add zlib support to Decompressor

2

Closes #1163. ### Changes - Add zlib support to `Decompressor`. - Add unit tests for zlib functionality. Python unfortunately doesn't have a file-like class for zlib, so I created my...

JohnHBrock

Does Collator need to exist?

1

### 📚 The doc issue Docs for [Collator](https://pytorch.org/data/0.6/generated/torchdata.datapipes.iter.Collator.html#torchdata.datapipes.iter.Collator) leave a lot of questions. > Collates samples from DataPipe to Tensor(s) by a custom collate function What does collate mean in...

lendle

### Changes - correct docstring of SampleMultiplexerDataPipe. The examples show a result of multiplexing 2 datapipes of length 5 but the two datapipes are initialized with 10 elements each

timothee1996

### 🚀 The feature An `IterDataPipe` which can consume from stdin and automatically re-cyle each epoch. ### Motivation, pitch I'd like to push data augmentation and preprocessing upstream so model...

erip

Dataloader2 with FullSyncIterDataPipe throws error during initilization

3

### 🐛 Describe the bug Hi, we found some strange during using Dataloader2. Here's some details about the issue. - We are a long run training job with 8 AWS...

chenxingyu-cs