PointRCNN inference issue

[Jaein94]

Checklist

• [X] I have searched for similar issues.
• [X] I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

My Question

hi. i'm trying to inference PointRCNN model. but problem occured while inference the model Below is the error:

code

import os
import open3d.ml as _ml3d
import open3d.ml.torch as ml3d
from open3d.ml.vis import Visualizer, BoundingBox3D, LabelLUT
from tqdm import tqdm
import time


cfg_file = "ml3d/configs/pointrcnn_kitti.yml"
cfg = _ml3d.utils.Config.load_from_file(cfg_file)

model = ml3d.models.PointRCNN(**cfg.model)
cfg.dataset['dataset_path'] = "/home/amlab/KITTI_point"
# cfg.dataset['dataset_path'] = "/home/amlab/vtd_kitti_format"
# cfg.dataset['dataset_path'] = "/home/amlab/os_test"

dataset = ml3d.datasets.KITTI(cfg.dataset.pop('dataset_path', None), **cfg.dataset)
pipeline = ml3d.pipelines.ObjectDetection(model, dataset=dataset, device="gpu", **cfg.pipeline)

# download the weights.
ckpt_folder = "./logs/"
os.makedirs(ckpt_folder, exist_ok=True)
ckpt_path = ckpt_folder + "pointrcnn_kitti_202105071146utc.pth"
# pointpillar_url = "https://storage.googleapis.com/open3d-releases/model-zoo/pointpillars_kitti_202012221652utc.pth"
# if not os.path.exists(ckpt_path):
#     cmd = "wget {} -O {}".format(pointpillar_url, ckpt_path)
#     os.system(cmd)

# load the parameters.
pipeline.load_ckpt(ckpt_path=ckpt_path)

test_split = dataset.get_split("training")
# print(test_split.__dict__)


vis = Visualizer()
lut = LabelLUT()
for val in sorted(dataset.label_to_names.keys()):
    lut.add_label(val, val)

# run inference on a single example.
# returns dict with 'predict_labels' and 'predict_scores'.
boxes = []
data_list = []
for idx in tqdm(range(10)):
    data = test_split.get_data(idx)
    true_val = data['bounding_boxes']
    result = pipeline.run_inference(data)[0]
    pred = {
    "name": 'KITTI' + '_' + str(idx),
    'points': data['point'],
    'bounding_boxes': result
    }
    data_list.append(pred)
    true = {
        "name": 'KITTI_true' + '_' + str(idx),
        'points': data['point'],
        'bounding_boxes': true_val
    }
    data_list.append(true)
    

vis.visualize(data_list, lut, bounding_boxes=None)

and got a issue.

Error

Traceback (most recent call last):
  File "test_main_point_rcnn.py", line 50, in <module>
    result = pipeline.run_inference(data)[0]
  File "/home/amlab/Open3D-ML_0.14/ml3d/torch/pipelines/object_detection.py", line 71, in run_inference
    results = model(data)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amlab/Open3D-ML_0.14/ml3d/torch/models/point_rcnn.py", line 120, in forward
    points)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amlab/Open3D-ML_0.14/ml3d/torch/models/point_rcnn.py", line 676, in forward
    x)  # (B, N, 3), (B, C, N)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amlab/Open3D-ML_0.14/ml3d/torch/modules/pointnet.py", line 95, in forward
    li_xyz, li_features = self.SA_modules[i](l_xyz[i], l_features[i])
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amlab/Open3D-ML_0.14/ml3d/torch/modules/pointnet.py", line 143, in forward
    new_features)  # (B, mlp[-1], npoint, nsample)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/amlab/anaconda3/envs/open3d/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [16, 3, 1, 1], expected input[1, 4, 4096, 16] to have 3 channels, but got 4 channels instead

[sanskar107]

@Jaein94 The problem seems to be that the KITTI pointcloud contain 4 dimensions (x, y, z, and intensity). Could you confirm this after adding data['point'] = data['point'][:, :3] in your for loop ?

[Jaein94]

Thank you! it is very helpful response! no more errors in the model. but i have additional question. i got some predtion results with kitti data set, but the model predict incorrect results. is there any advice? below is the prediction results

[martinteh]

Could you show your resulting code after including that in the for loop?