Add a new pointcloud filter based on image semantic segmentation

[badai-nguyen]

Checklist

• [X] I've read the contribution guidelines.
• [X] I've searched other issues and no duplicate issues were found.
• [X] I've agreed with the maintainers that I can plan this task.

Description

As I described https://github.com/orgs/autowarefoundation/discussions/3974, we (TIER IV) and other teams are also facing difficult to deal with unknown from plants, raindrops/smoke/splash pointcloud or pointcloud of ground segmentation fail.

Current pointcloud filters including dual_return_filter, outlier_filter, ground_segmentation only based on points kinematics information. Compare_map_filter compare pointcloud with map to remove un-movable objects but it frequently fail with growing plants, tree.. etc.

It should be helpful if image semantic information could be used and fused with pointcloud to remove unnecessary pointclouds for autonomous driving such as pointcloud of plants in background, or drivable area.

Purpose

To deploy some kind of image semantic or instance segmentation model and matching with pointcloud to filter pointcloud.

Possible approaches

1. Train and deploy an image segmentation model:

• By Using a separate model with current detection Yolox model
• Or adding an additional task/header for into current detection yolox model

2. Publish some 2d mask image of segmentation information or kind of less important image region information for filtering

3. Create new node to fuse image pixel level with pointcloud and remove pointcloud in less important image region.

Definition of done

• [ ] Deploy an image segmentation into autoware and
• [ ] Publish some topics of image segmentation result
• [ ] Matching pointcloud and image segmentation result for filtering

[stale[bot]]

This pull request has been automatically marked as stale because it has not had recent activity.

[xmfcx]

• Retraining the 2D image segmentation model right now.
• Semantic seg fusion is already merged.
• Will update the pipeline to have this included.
• Test on the real vehicle.
• Eval on old data (e.g. noisy)

[xmfcx]

@StepTurtle please check this work too, link it when creating a new issue.

[badai-nguyen]

Note: 2D multiple header YOLOX was trained and is going to update by https://github.com/autowarefoundation/autoware/pull/4012