Abnormal demo.py result using nuscenses dataset pre-trained models

[yaobaishen]

This issue is related to issue 1341, but here we track a different problem. After modify the demo.py a bit in order to run the nuscenses dataset pre-trained model, I get weird inference result. The modification is below, because the nuscenses lidar data has 5 dimensions: ['x', 'y', 'z', 'intensity', 'timestamp'].

    def __getitem__(self, index):
        if self.ext == '.bin':
            if self.is_nuscense_data:
                points = np.fromfile(self.sample_file_list[index],
                                     dtype=np.float32).reshape(-1, 5)
            else:
                points = np.fromfile(self.sample_file_list[index],
                                     dtype=np.float32).reshape(-1, 4)

And the command line to test CenterPoint model with demo.py. Note: I have added a --nuscense_data flag, and test with a xxx.pcd.bin file from nuscenes_mini.

python demo.py --cfg_file cfgs/nuscenes_models/cbgs_dyn_pp_centerpoint.yaml --ckpt ../model_zoo/nuscenes_models/cbgs_pp_centerpoint_nds6070.pth --data_path dataset/nuscenes_mini/samples/LIDAR_TOP/n015-2018-11-21-19-38-26+0800__LIDAR_TOP__1542801004447480.pcd.bin --nuscense_data

Here is the inference result, even after I set a score threshold 0.3, there are many abnormal detection boxes:

Could anyone share some insights about how to use the nuscenses dataset pre-trained models? Thanks a lot.

[yaobaishen]

I also try the PointPillar model download from the main-page, the inference result still looks strange. command line:

python demo.py --cfg_file cfgs/nuscenes_models/cbgs_pp_multihead.yaml --ckpt ../model_zoo/nuscenes_models/pp_multihead_nds5823_updated.pth --data_path dataset/nuscenes_mini/samples/LIDAR_TOP/n015-2018-11-21-19-38-26+0800__LIDAR_TOP__1542801004447480.pcd.bin --nuscense_data

visulization:

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[muvanpersie]

Hi, have you solve the probelm yet? I find that the format of nuscenes lidar "bin" file is [x, y, z, intensity, ring_id], the last channel is ring_id but not timestamp. So I think we should drop the last "ring_id" channel and concat with the right "timestamp".

[yaobaishen]

@muvanpersie thanks for update, so how about your result looks like? I haven't look into this project for a long while.

[muvanpersie]

@muvanpersie thanks for update, so how about your result looks like? I haven't look into this project for a long while.

Much better but with a few false alarms after simply replace the DemoDataset by NusencesDataset. And for 'bin' files, we should transform the other 9 sweeps frame to the 1 sample frame, with the time difference as the fifth "timestamp" channel.

[yaobaishen]

Sorry for late update. Thanks for @muvanpersie 's tips, after replace the DemoDataset by NusencesDataset, and create the nuscenes_infos_10sweeps_train.pkl / nuscenes_infos_10sweeps_val.pkl following the official guide, the visualization of demo.py looks much better now.

[HHX0607]

Hi! @yaobaishen I want to infer and visualize result from demo.py. I use : python demo.py --cfg_file cfgs/nuscenes_models/cbgs_voxel0075_res3d_centerpoint.yaml --ckpt ../ckpt/voxelnext_nuscenes_kernel1.pth --data_path ../../data/Nuscenes/v1.0-test/samples/LIDAR_TOP/n008-2018-08-01-15-34-25-0400__LIDAR_TOP__1533152214547215.pcd.bin Can you tell me exactly what changes I need to make on demo.py, I've already created the relevant .pkl based on the guide, but it's not clear how to input the point cloud (kitti model just passes directly into a single .bin ).

[yaobaishen]

@HHX0607 You only need to change the DemoDataset to NuScenesDataset, just like what muvanpersie paste in https://github.com/open-mmlab/OpenPCDet/issues/1627#issuecomment-2757329839 Then the demo.py will process the data files specified in cfg.DATA_CONFIG, you don't need to specify the bin files in command line. I use the demo.py like this: python demo.py --cfg_file cfgs/nuscenes_models/cbgs_second_multihead.yaml --ckpt ../model_zoo/nuscenes_models/cbgs_second_multihead_nds6229_updated.pth

[Dev-HJYoo]

@yaobaishen Hi! i want to infer and visualize on nuscenes dataset.

I replaced the demo dataset with the nuScenes dataset as instructed, and generated nuscenes_infos_10sweeps_train.pkl and nuscenes_infos_10sweeps_val.pkl following the official guide.

However, when testing the SECOND model, there were too many false positive detections, as shown in the figure below. Could you please help me figure out what might be causing this issue?

My demo.py

import argparse
import glob
from pathlib import Path

from pcdet.datasets.nuscenes import nuscenes_dataset

try:
    import open3d
    from visual_utils import open3d_vis_utils as V
    OPEN3D_FLAG = True
except:
    import mayavi.mlab as mlab
    from visual_utils import visualize_utils as V
    OPEN3D_FLAG = False

import numpy as np
import torch

from pcdet.config import cfg, cfg_from_yaml_file
from pcdet.datasets import DatasetTemplate
from pcdet.models import build_network, load_data_to_gpu
from pcdet.utils import common_utils
from pcdet.datasets import NuScenesDataset

def parse_config():
    parser = argparse.ArgumentParser(description='arg parser')
    parser.add_argument('--cfg_file', type=str, default='cfgs/kitti_models/second.yaml',
                        help='specify the config for demo')
    parser.add_argument('--data_path', type=str, default='demo_data',
                        help='specify the point cloud data file or directory')
    parser.add_argument('--ckpt', type=str, default=None, help='specify the pretrained model')
    parser.add_argument('--ext', type=str, default='.bin', help='specify the extension of your point cloud data file')

    args = parser.parse_args()

    cfg_from_yaml_file(args.cfg_file, cfg)

    return args, cfg


def main():
    args, cfg = parse_config()
    logger = common_utils.create_logger()
    logger.info('-----------------Quick Demo of OpenPCDet-------------------------')
    demo_dataset = NuScenesDataset(
        dataset_cfg=cfg.DATA_CONFIG,
        class_names=cfg.CLASS_NAMES,
        training=False,
        logger=logger
    )
    logger.info(f'Total number of samples: \t{len(demo_dataset)}')

    model = build_network(model_cfg=cfg.MODEL, num_class=len(cfg.CLASS_NAMES), dataset=demo_dataset)
    model.load_params_from_file(filename=args.ckpt, logger=logger, to_cpu=True)
    model.cuda()
    model.eval()
    with torch.no_grad():
        for idx, data_dict in enumerate(demo_dataset):
            logger.info(f'Visualized sample index: \t{idx + 1}')
            data_dict = demo_dataset.collate_batch([data_dict])
            load_data_to_gpu(data_dict)
            pred_dicts, _ = model.forward(data_dict)

            V.draw_scenes(
                points=data_dict['points'][:, 1:], ref_boxes=pred_dicts[0]['pred_boxes'],
                ref_scores=pred_dicts[0]['pred_scores'], ref_labels=pred_dicts[0]['pred_labels']
            )

            if not OPEN3D_FLAG:
                mlab.show(stop=True)

    logger.info('Demo done.')


if __name__ == '__main__':
    main()