Sebastian Hoffmann

For my specific use case `.repeat(n_workers).sharding_round_robin_dispatch()` can be used for now as a workaround (I believe).

No this workaround actually does not work, as this example demonstrates: ``` my_worker_info = None def abc(x): return f'Worker {my_worker_info.worker_id}: {x}' def worker_init(dp, worker_info): global my_worker_info my_worker_info = worker_info return...

I created a small proof of concept and benchmarked it: https://github.com/sehoffmann/numpy_shared_mem/blob/master/numpy_shared_memory.ipynb Bottom line: A order of magnitude faster. Also, if preloading the memory into shared memory, it obviously won't be...

Also, I did some small testing and it appears that this new serialization method for `np.ndarray` also, out-of-the-box, trickles down to downstream libraries such as xarray. I.e. these libraries and...

I did some further testing and can confirm these points: * The approach directly works with both `xr.DataArray` as well as `xr.Dataset`, containing multiple variables and coordinate arrays * When...

I can report that in a real pipeline with real data implementing this results in a 13.2% throughput increase. But obviously the biggest gain is the vastly reduced memory consumption...

Hey @ejguan, I'm already solely working with the Dataloader2. Also, since at this point my current implementation does all IPC via shared mem, I experimented with transmitting the processed arrays...

> > This resulted in a degradation of the performance to the single-threaded case which lets me believe that my main performance overhead right now is actually the `collate`. >...

On a related note: I find the check at https://github.com/pytorch/pytorch/blob/master/torch/utils/data/graph_settings.py#L53 overly restrictive, and there should be an optional way to disable it in my opinion. I already ran into it,...

> 1. Clearly document (if allowed) the how + expectations of implementing custom `shuffle()`, `apply_sharding()` (and other...) methods. Currently, this is hard to do without actually reading the `DataLoader` source...