POC semantic segmentation

[chingyulin]

This PR is intended to open a discussion rather than a merging-ready PR, although most of the heavy-lifting: OpenGL, math has implemented.

https://user-images.githubusercontent.com/60384727/177585341-b5b9bdca-99b8-40e6-8936-4c12533f67d0.mov

This PR introduces a bare minimum requirement to assign point-level labels using 3D bounding boxes for training semantic segmentation models. Is this something of interest?

If so, there are some open questions:

• How do we store/ load point-labels? Open3D PointCloud class doesn't support label storage? As a separate numpy file?
• Switch between object detection and semantic segmentation modes or use the same UI for both tasks
• To store and color the points, we need to know the distinct label upfront but currently, the box class is a free edit box. Should we change it to a drop-down list where users can define the labels in config.ini?
• How should the UI look like for the semantic segmentation mode. Below is my suggested UI, the notable differences are as follows:

1. Three more buttons for swapping colors, resetting the labels and saving the labels
2. A color list of labels and the number of points in each label
3. Change the box label list to a drop-down list
4. Two more buttons to assign points to the boxes if they are inside them.

Please let me know what you think :)

Suggested UI

[ch-sa]

Hi @chingyulin,

thank you for the PR and sorry for the late response (I wanted to look at the code first but did not find the time yet!).

In general: I like the idea of supporting point segmentation via bounding boxes.

I think the first step should be supporting the export via postprocessing and optional coloring.

Can you name common label formats for point segmentation?

For the further steps, I think they should not bloat the normal object detection (bounding box only) mode if they don't add value there.

However, we could show these controls only on activation of a "segmentation mode".

What do you think.

(Will reply to the code as soon as I find the time!)

Best,

Christoph

[chingyulin]

Hi @ch-sa,

Thanks for the response.

Regarding the common label formats for point segmentation, I did a quick search (SemanticKitti, S3DIS) and they don't use the same label format. In Open3D-ML, they represent labels together with the point cloud features (positions and colors) as a numpy array. I suggest we can follow this format so the labeled result can be easily used together with Open3D-ML. What do you think? If it's ok for you, I can start the implementation.

I agree that we should not float the normal bounding box mode. I can make a field in the config to activate the "segmentation mode".

Best,

Ching-yu

[ch-sa]

Hi again,

I tried your branch and works already quite nice. From your code I see that you create an additional "label color" VBO so the original colors remain untouched, which is good and needed in case a user aligns and exports a point cloud.

As this a rather large change I created a new feature branch feature/semantic-segmentation and would propose that you merge into this one first. This way you can provide smaller step-by-step pull requests and I could support you along the way.

So far you used a button for switching to the segmentation mode. I would rather see it as an option in the menu that can be activate/ checked, like this:

For the export format, I agree that we can start with that format (we can still add more later).

Simply add a new label manager (writers/ readers) by extending the Base: https://github.com/ch-sa/labelCloud/blob/master/labelCloud/io/labels/base.py

Some general hints for the code conventions:

1. relative imports over absolute imports
2. black formatting
3. type hinting of parameters and return values

Best and looking forward,

Christoph

[chingyulin]

close this one as it's been discussed.