Fix bug of multiple pre-processing when segmentation (PyTorch)

[Lionelsy]

It is very slow in performing segmentation inference. #531 #234

And, it is because the dataloader will apply multiple data preprocessing if self.cache_convert is None. https://github.com/isl-org/Open3D-ML/blob/fcf97c07bf7a113a47d0fcf63760b245c2a2784e/ml3d/torch/dataloaders/torch_dataloader.py#L77-L83

When running the run_inference method, the cache_convert of dataloader is None. https://github.com/isl-org/Open3D-ML/blob/fcf97c07bf7a113a47d0fcf63760b245c2a2784e/ml3d/torch/pipelines/semantic_segmentation.py#L143-L147

This leads to extreme slowness in performing reasoning.

I've added a get_cache method to provide cache to avoid slowdowns caused by multiple preprocessing during inference.

I tested it on a GV100 GPU with RandLA-Net on the Toronto3D dataset. Inferencing time for a single scene is only two minutes and 37 seconds. Reasoning is considerably faster than before

After: test 0/1: 100%|██████████████████████████████████████████████████████| 4990714/4990714 [02:37<00:00, 31769.86it/s]

Before: test 0/1:   4%|██                                                     | 187127/4990714 [05:12<2:19:39, 573.27it/s]

[rejexx]

I applied this fix on a local fork with a custom dataset and saw RandLA-NET inference go from 27 hours to 6 minutes. I can't thank you enough for sharing.

[ssheorey]

Hi @Lionelsy thanks for debugging this and submitting a PR. I have a question:

• if cache_convert is None, it looks like preprocess is applied only once in line 81 of torch_dataloader.py. Can you point out the multiple pre-processing?

[Lionelsy]

Hi @Lionelsy thanks for debugging this and submitting a PR. I have a question:

• if cache_convert is None, it looks like preprocess is applied only once in line 81 of torch_dataloader.py. Can you point out the multiple pre-processing?

Thank you for your continued contributions to Open3D!

In fact, the __getitem__ method in the torch_dataloader.py is called multiple times during the inference process, and the preprocess method (a very time-consuming step) is called again each time.

If we use the cache_convert to store the preprocessed data, it will save much time.