Philip Meier

I think I found the issue. The problem is that the `current_batch` stays `0` after the first call https://github.com/libffcv/ffcv/blob/f25386557e213711cc8601833add36ff966b80b2/ffcv/memory_managers/process_cache/schedule.py#L113-L114 while `self.next_batch` gets incremented: https://github.com/libffcv/ffcv/blob/f25386557e213711cc8601833add36ff966b80b2/ffcv/memory_managers/process_cache/schedule.py#L129-L131 Going up the trace, we see...

@rmrafailov Looking at your code, the memory leak could come from the fact that you are creating the iterator over and over. Compare ```py for _ in range(1000): batch =...

Test failure is unrelated.

> reproducing the code outside of a notebook to see if it still fails You can do this by running the following in your VSCode terminal ```sh python -c 'import...

Could you clarify what you mean by "read videos from memory"? Do you want to a dataset that allows you to read data from a LMDB? If yes, you can...

> Oh interesting so the way you'd integrate new backends in the future is to integrate them within `torch.data`? Not sure what you mean by "backends" here. In general you...

@msaroufim > to clarify by backend I mean one of these https://github.com/pytorch/vision#image-backend - i.e: pillow, accimage, pillow simd etc.. The new datasets will return a `features.EncodedImage`, which is a 1D...

Same for segmentation: https://github.com/pytorch/vision/blob/cac4e228c9ca9e7564cb34406e7ebccfdd736976/references/segmentation/train.py#L84

This might be a good onboarding task for the new datasets.

@23pointsNorth We are currently halting further contributions to the prototype datasets, since there are some details that need to be figured out. If that is resolved, I'm guessing @NicolasHug can...