isl-org
Can't pickle local object while training RandLANet on S3DIS
Checklist
- [X] I have searched for similar issues.
- [X] I have tested with the latest development wheel.
- [X] I have checked the release documentation and the latest documentation (for
masterbranch).
Describe the issue
Can't pickle local object while training RandLANet on S3DIS. I use pytorch.
Steps to reproduce the bug
import open3d.ml as _ml3d
import open3d.ml.torch as ml3d
model = ml3d.models.RandLANet()
dataset_path = "/Users/kimd999/research/projects/Danny/files/public_dataset/S3DIS/Stanford3dDataset_v1.2_Aligned_Version"
dataset = ml3d.datasets.S3DIS(dataset_path=dataset_path, use_cache=True)
pipeline = ml3d.pipelines.SemanticSegmentation(model=model, dataset=dataset, max_epoch=100)
# prints training progress in the console.
pipeline.run_train()
Error message
INFO - 2022-02-15 13:24:10,927 - semantic_segmentation - DEVICE : cpu
INFO - 2022-02-15 13:24:10,927 - semantic_segmentation - Logging in file : ./logs/RandLANet_S3DIS_torch/log_train_2022-02-15_13:24:10.txt
INFO - 2022-02-15 13:24:10,929 - s3dis - Found 249 pointclouds for train
INFO - 2022-02-15 13:24:10,935 - s3dis - Found 23 pointclouds for validation
INFO - 2022-02-15 13:24:10,937 - semantic_segmentation - Initializing from scratch.
INFO - 2022-02-15 13:24:10,940 - semantic_segmentation - Writing summary in train_log/00003_RandLANet_S3DIS_torch.
INFO - 2022-02-15 13:24:10,940 - semantic_segmentation - Started training
INFO - 2022-02-15 13:24:10,940 - semantic_segmentation - === EPOCH 0/100 ===
training: 0%| | 0/63 [00:00<?, ?it/s]
Traceback (most recent call last):
File "train_model_for_semantic_segmentation.py", line 19, in
Expected behavior
No response
Open3D, Python and System information
- Operating system: OSX 10.15.7
- Python version: Python 3.8.12
- Open3D version: open3d version:0.14.1
- System type: x84
- Is this remote workstation?: no
- How did you install Open3D?: pip install open3d
Additional information
No response
I am having the exact same issue, also with RandLANet and SemanticSegmentation. I will let you know if I find the problem @kimdn .
@kimdn Seems that this was caused somehow by num_workers in pytorch, see this thread: https://github.com/pyg-team/pytorch_geometric/issues/366.
Try setting num_workers=0 in your pipeline definition like so:
pipeline = ml3d.pipelines.SemanticSegmentation(model=model, dataset=dataset, max_epoch=100, num_workers=0)
I guess it is not a great solution if you intend to have num_workers > 0, but hopefully it will at least resolve the error message!
I used WSL ubuntu to train the models. Num_workers > 0 worked for RandLA-Net but KPConv, which was very strange. But at least it proved that multiprocessing could work in this virtual environment. Do you have any ideas about the difference in the model deployments?
Hi @bernhardpg @LuZaiJiaoXiaL
I have set num_workers to 0, but I still met this bug. Do you know how to solve? python scripts/run_pipeline.py torch -c ml3d/configs/randlanet_toronto3d.yml --dataset.dataset_path dataset/Toronto_3D --pipeline SemanticSegmentation --dataset.use_cache True --num_workers 0
INFO - 2022-12-09 17:31:29,220 - semantic_segmentation - === EPOCH 0/200 ===
training: 0%| | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/export/home2/hanxiaobing/Documents/Open3D-ML-code/Open3D-ML/scripts/run_pipeline.py", line 246, in
Hey, any new insights into how to fix the problem? I just ran into the same issue on a dockerized ubuntu20.04 with cudnn11.7.
Happy, if someone could share their latest fixes
Hi, I find a solution. Just add "num_workers:0 pin_memory: false" belong to "pipeline" in ".yaml " config file. Solution link https://blog.csdn.net/weixin_40653140/article/details/130492849
