Need some instruction for python api development (PyDetector and PyMatcher)

[cdb0y511]

Hi, @matlabbe. Recently, I noticed some nice matchers (based on pytroch) and detectors on github, which archive the state-of-art. Like https://github.com/cvg/LightGlue, I think it is better than superGlue, at least faster. So, I am trying to embed it into rtabmap, following your rtabmap_superGlue.py. However, I need help with C++ and Python mixed debugging. I followed your advice using docker with the docker file frontiers2022.

1. Maybe you can provide a pure Python example or test example for Pydetect, Help me understand how the input feature format (I find it has C++ version superpoint detect) can be used for Pydetect and how the matcher returns.

2. I find it may not have fully developed the Pydetect for feature detection (for Matcher only, but some recent features like DISK can also be embedded into rtabmap via Python). I think the Python detected and matcher may improve results, cause the vis part is the most important one in rtabmap, and the torch-based matcher and detectors are trending. I am looking forward to making PR if they work well. Thanks,

[cdb0y511]

Developers are interested in the recent pytorch-based matcher and detector. I hope @matlabbe can give the Python API soon. A pure Python test or example file is fine, and I have trouble in C++ and Python mixture development; I have to guess the input or return for Python scripts. Or @matlabbe has a better way to let us embed the state-of-the-art pytorch-based methods.

[matlabbe]

There is currently two examples for feature matching using python:

• https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superglue.py
• https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_oanet.py and another example for feature extraction:
• https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superpoint.py

For debugging python scripts called by cpp, it is painful. Currently, you may try on of these examples and print the structure of matches returned and data sent form cpp. I have currently some issues when python code is not running in main thread. I think as you suggested, having a Python API for rtabmap (where we call cpp from python) would be better for python/ML folks!

When you are referring Pydetect, can you add link to make sure I am looking at the right project?

[cdb0y511]

Hi,@matlabbe. For this issue https://github.com/introlab/rtabmap/issues/896#issuecomment-1575742541, I have to move the whole development environment into docker with a docker file similar to https://github.com/introlab/rtabmap/blob/master/docker/frontiers2022/Dockerfile.

Turning into call cpp from Python is good, but I don't know how much work you should do. Any timeline?

The vis part (feature detect and matcher) can now utilize the state-of-the-art ML work.

I found the Visual Registration has PyMatcher(https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superglue.py) and PyDetector (https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superpoint.py) So, I want to start with it. I also found that superpoint detect is written in native C++ code (https://github.com/introlab/rtabmap/blob/master/corelib/src/superpoint_torch/SuperPoint.cc). So lightGlue can utilize the detector; some simple work let us try lightGlue is to modify the (https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superglue.py). But I don't know the input for Python and return; maybe the classic way is to print the data structure.

[cdb0y511]

Also, I think some documents about Python input and return (https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superglue.py, https://github.com/introlab/rtabmap/blob/master/corelib/src/python/rtabmap_superpoint.py)) would be nice; this will save a lot of time for latecomers. If @matlabbe has the time and wants to expand the vis part (focus on visual registration) in Python.