It seems the dynamic objects not been tracked

[XunshanMan]

hi, thanks for the fantastic work. I met the similar problem like https://github.com/martinruenz/maskfusion/issues/21.

I did according to the REAME.md, the only difference is that I added two CMake parameters as below, so that there would not be "out of memory" error on my GPU ( Nvidia 840M, yes... an old one)

-DMASKFUSION_NUM_GSURFELS=921600 -DMASKFUSION_NUM_OSURFELS=102400

and checked my toml file to keep it unchanged:

# If not empty, ignore every class not listed here. Example: ['teddy bear']
filter_classes = []

# If not empty, only track classes listed here (otherwise apply BG motion)
trackable_classes = []

# Reject predictions with lower score
score_threshold = 0.55

However, when running on the dataset from Co-Fusion, it seems the label "clock" has been filtered.

so in the final result, the trajectory of the clock makes the reconstruction not so good:

The offline result from MaskRCNN shows good:

I'm wondering why the dynamic parts are ignored and not tracked. Is there some size filter somewhere else? My output shows like below:

Initialised multi-object fusion (main-thread: 140037120672128)
- The background model can have up to 921600 surfel (960x960)
- Object models can have up to 102400 surfel (320x320)
- Using GPU 0 for SLAM system and GPU 0 for MaskRCNN
- Using frame-queue of size: 0
New label detected (**2147483647,2147483647 -2147483648,-2147483648**) - try relocating...
Found new model.
Created model with max number of vertices: 102400
New label detected (2147483647,2147483647 -2147483648,-2147483648) - try relocating...
Found new model.
Created model with max number of vertices: 102400

"2147483647,2147483647 -2147483648,-2147483648" is a little weird, however I checked the codes and they are defined as initial values for bounding boxes.

Thanks so much. I'll keep on digging into the codes to find the answer.

[CharlieLeee]

@XunshanMan Hi, would you mind helping me with the issue, and I would really appreciate it if you could tell me how to generate the result shown in your issue:

[XunshanMan]

Hi, @CharlieLeee It's the result of the offline mode.

Running MaskRCNN offline, before executing MaskFusion, and try the command as shows in the README.md ./offline_runner.py -i /path/to/rgb/frames -o /path/to/output/masks --filter teddy_bear

[CharlieLeee]

@XunshanMan Thank you so much for replying! I'm running the maskrcnn right now, I just want to make sure that we need to convert klg to rgb first right? like this: ./convert_klg -i /home/charlie/Downloads/sliding-clock.klg -o /home/charlie/Downloads/output_mask -frames -tum

When I'm running maskrcnn, the output files include three files: jpg, png, and txt. It's weird that the png file looks like there's no mask in it

[XunshanMan]

@CharlieLeee yes, we need convert klg to png first.

For why it seems no mask in the pngs, I think every pixels have a small integer value to represent a label, e.g. 0 for human beings, 1 for some object, 2 for another object.... The type of the png may be 16UC1, so the maximum value is 65535, and the area of teddy bear may be too small compared with the maximum, so it is near black. But not totally black. In fact if you see the 0000.png carefully, there is a teddy bear. You could also try outputing its value using opencv.

[CharlieLeee]

@XunshanMan Yeah! you're right, I totally ignore the mask of the teddy bear haha.

So after generating the mask, all I need is to collect all pngs in a mask folder and run the command: ./MaskFusion -run -l /home/charlie/Downloads/teddy-handover.klg -maskdir /path/to/all/generated/png is that right?

Updates:

I ran the command ./offline_runner.py -i /home/charlie/Downloads/output_mask/rgb/ -o /home/charlie/Downloads/output_mask/rcnn/ --filter teddy_bear first and then collect all the .png mask files as well as the .txt files into a folder named masks

then I ran ./MaskFusion -run -l /home/charlie/Downloads/sliding-clock.klg -maskdir /home/charlie/Downloads/output_mask/masks/ but unfortunately, there's still no segmentation result appeared.

Could you please suggest which step I did wrong, thanks!

[XunshanMan]

@CharlieLeee Hi, it seems there are some parameters you could adjust, for example, the "Threshold". And if you want to track those objects, open the option "Track all models". I opened this issue two month ago so I couldn't remember very clearly. There should be no differences whether you run the Mask-RCNN online or offline... I think the parameters are the key. Wish you good luck!

[CharlieLeee]

@XunshanMan Sure, I'll try that! Thank you so much for helping!!