TypeError: h5py objects cannot be pickled

[liuyyy111]

hi, I am trying to run train.py, I get an error as "TypeError: h5py objects cannot be pickled". I think the error is stemming from torch.utils.data.DataLoader. or did i miss any configurations? i have no idea about how to tackle it.

the error stack trace is as follow.

Traceback (most recent call last): File "D:/study/workspace/Python/a-PyTorch-Tutorial-to-Image-Captioning/train.py", line 333, in main() File "D:/study/workspace/Python/a-PyTorch-Tutorial-to-Image-Captioning/train.py", line 117, in main epoch=epoch) File "D:/study/workspace/Python/a-PyTorch-Tutorial-to-Image-Captioning/train.py", line 164, in train for i, (imgs, caps, caplens) in enumerate(train_loader): File "D:\Environment\Anaconda\envs\PyTorch\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter return _MultiProcessingDataLoaderIter(self) File "D:\Environment\Anaconda\envs\PyTorch\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init w.start() File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\popen_spawn_win32.py", line 89, in init reduction.dump(process_obj, to_child) File "D:\Environment\Anaconda\envs\PyTorch\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) File "D:\Environment\Anaconda\envs\PyTorch\lib\site-packages\h5py_hl\base.py", line 308, in getnewargs raise TypeError("h5py objects cannot be pickled") TypeError: h5py objects cannot be pickled

Process finished with exit code 1

[Christine620]

hello,I face the same problems with you? how can i handle this problems?

[JerryX1110]

@liuyyy111 @Christine620 It happens because h5py won't read from multiple processes. By omitting num_workers, you're setting it to the default of 0, which uses only the main process. You could also set it to 1 (on Linux, at least). So try to set :

num_workers=0. it works for me.