Vitaly Fedyunin
Vitaly Fedyunin
> Are we considering `PinMemory` and `Shuffle` (to turn on and off) as adapters as well? Or those will be strictly DataPipes? They should be both adapters and pipes.
@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff [on Phabricator](https://www.internalfb.com/diff/D36866290).
@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff [on Phabricator](https://www.internalfb.com/diff/D36866290).
Looks like pin_memory should be a parameter of ReadingService, otherwise 'pin' gets lost when tensors are moved from child processes to the main training loop. CC @ejguan
Makes total sense to me, I will update MPReadingService to respect such DataPipes as part of this PR.
Looks like `dill` trying to pickup the entire torch module (or a smaller part of it). One of the options is to make functions in the test pickable, or move...
Can we derive `total_length` from source Datapipe if possible?
I think question here is how big is the pickled data. If it fits memory nicely (I presume it just have too, due to the Pickle nature) the best case...
Note: If we allow such DataPipe we would need to take special care of order nondeterministic nature of most SQL database engines.