nestedtensor icon indicating copy to clipboard operation
nestedtensor copied to clipboard

Cannot use nestedtensor in Dataloader?

Open justanhduc opened this issue 6 years ago • 1 comments

I got a pickler error in multiprocessing that it cant pickle the nestedtensor object. does it mean i cant use nestedtensor for Dataset? I guess using nestedtensor in Dataset is important as nestedtensor always copies tensors, which will double the memory usage.

Traceback (most recent call last):
  File "/home/justanhduc/anaconda3/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/home/justanhduc/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle nestedtensor._C._BufferNestedTensor objects

justanhduc avatar Dec 13 '19 15:12 justanhduc

We'll add pickling support once _BufferNestedTensor (and possibly NestedTensor) has been moved into C++.

cpuhrsch avatar Dec 13 '19 22:12 cpuhrsch