nestedtensor
nestedtensor copied to clipboard
Cannot use nestedtensor in Dataloader?
I got a pickler error in multiprocessing that it cant pickle the nestedtensor object. does it mean i cant use nestedtensor for Dataset? I guess using nestedtensor in Dataset is important as nestedtensor always copies tensors, which will double the memory usage.
Traceback (most recent call last):
File "/home/justanhduc/anaconda3/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
obj = _ForkingPickler.dumps(obj)
File "/home/justanhduc/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
TypeError: can't pickle nestedtensor._C._BufferNestedTensor objects
We'll add pickling support once _BufferNestedTensor (and possibly NestedTensor) has been moved into C++.