Android Studio setup

[SaurabhMishra2003]

Hi community, Can you all please guide me, how can I set up this project in Android studio. Basic cloning is not helping because of missing gradle files.

[matlabbe]

I don't have a guide to do so. The code is CMake-based instead of gradle approach. The android project is not actively developed anymore (moved most effort on iOS), but if I need to do changes on java part, I'll still use Eclipse (with android plugin for UI layout changes) and open the project, while still compiling using cmake/make in a side terminal.

If someone is motivated to share a Gradle/Android studio project config that works, it would be great. I think the hardest part is to build all dependencies for android. To do so, we can look at this docker image that I used for android development: https://github.com/introlab/rtabmap/tree/master/docker/bionic/android

[SaurabhMishra2003]

Hi @matlabbe ,

I have tried it and was successful to run the project through Android studio. I reverse engineered the APK and got to know the needed libraries required for it. Then compiled it through docker. Created necessary build files for Android studio, manually (it was tricky, but got through it). Placed precompiled libraries in their respective position (jniLibs) so that they are shared while creating Android package. Its running well.

But here I have a question now. Suppose, I need to improve the mapping now, so need access to those source files. So how do I make changes in those file (algorithms and mathematical part) ? Do I need to make changes and again compile them in docker to get the libraries (.so files) ? I need your assistance here. I m very close.

Thanks, Saurabh

[matlabbe]

From what I understand, you are now able to re-build only code inside that folder: https://github.com/introlab/rtabmap/tree/master/app/android, but not the whole library or dependencies.

The "easy" way to rebuild rtabmap libraries is to cd rtabmap and do:

docker build -f docker/noble/android/rtabmap_apiXX/Dockerfile --build-arg API_VERSION=30 -t rtabmap_android .

To rebuild dependencies:

docker build -f docker/noble/android/Dockerfile -t introlab3it/rtabmap:android-noble-deps .

Note that if I am going to update/modify rtabmap library, I would use the android dev container available when you open rtabmap repo in vscode (note that I just updated it to work with latest vscode). As shown at the end of the dev container startup terminal, open a terminal and do:

export ANDROID_API=30 
cd build_android/arm64-v8a 
cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake \
   -DANDROID_ABI=arm64-v8a \
   -DANDROID_NDK=$ANDROID_NDK \
   -DANDROID_NATIVE_API_LEVEL=$ANDROID_API \
   -DBUILD_SHARED_LIBS=OFF \
   -DCMAKE_BUILD_TYPE=Release \
   -DCMAKE_INSTALL_PREFIX=/opt/android/arm64-v8a \
   -DCMAKE_FIND_ROOT_PATH="/opt/android/arm64-v8a/bin;/opt/android/arm64-v8a;/opt/android/arm64-v8a/share" \
   -DBUILD_EXAMPLES=OFF \
   -DBUILD_TOOLS=OFF \
   -DOpenCV_DIR=/opt/android/arm64-v8a/sdk/native/jni \
   ../..

make -j6

to build rtabmap library and the APK at the same time.

[SaurabhMishra2003]

Hi @matlabbe

For my android phone, current repo is obviously not taking depth data in input. But is it possible to use depth data, maybe through ARCore Depth API ? Pass it into input data and also store it in .db file ! How can I achieve this ?

Thanks, Saurabh

[matlabbe]

The current android app can do that, but the results are pretty bad (in terms of 3D reconstruction accuracy even if the trajectory is okay).

To try it, at least android26 is required if I remember well, enable "Depth From Motion" under the Mapping parameters, then on main settings page, select ARCore NDK camera driver.

[SaurabhMishra2003]

@matlabbe Thank you for your quick response.

I tried with above parameters. Results are actually not that bad, its decent. But yes, there are some gaps in 3D reconstruction. Attaching a video for your reference.

https://github.com/user-attachments/assets/3c2c5e70-2295-4e37-9653-a7b5f4c43814

Now, How can I improve the reconstruction/mapping part ? Also, can I increase the resolution ? I have already enabled "HD Mode" under Mapping parameters.

Thank you. Saurabh

[matlabbe]

By resolution, do you mean more points/polygons or high resolution better textures? The HD mode only increases the resolution of the RGB image.

For the geometric accuracy of the point cloud, without a TOF camera, it will be always distorted. There is not a lot we can do to improve it with the current approach.

For RGB-only reconstruction, I suggest to look at photogrammetry/Structure-From-Motion Apps, they will give a lot more accurate dense reconstruction.

[SaurabhMishra2003]

Hi @matlabbe

For RGB image resolution, even if I enable 'HD mode' : image resolution is 480 * 640 (checked in created .db file). Whereas, in iOS, if I enable 'HD mode', resolution of image is 1440 * 1920. Can we increase it for Android ?

Also, I have few doubts. I need your guidance on that. (for context : We are using ARCore ndk as camera driver and 'Depth from Motion' is ON; you can refer above videos for the current results)

1. For better reconstruction, can we increase FPS ? Maybe more FPS reduces chances of drift. If yes, where can I change the FPS ?
2. I guess other questions depends on your above answers haha :)

[matlabbe]

On Android, maybe ARCore cannot provide higher resolution images. Looking at the code, there is this part commented: https://github.com/introlab/rtabmap/blob/e5655b0b961df83a94b4fac423584ed68dcd3d18/app/android/src/com/introlab/rtabmap/ARCoreSharedCamera.java#L396-L401 Don't remember what was the issue when trying to use higher resolution settings, it looks like there was an error. More debugging would be required to see if it could work.

ARCore is already running at maximum FPS (I think), if you see drift, it is coming from ARCore.

[SaurabhMishra2003]

Hi @matlabbe , How are you ?

Is the IMU data already fused in this flow ? I mean I'm using this Android non-lidar with ARCore depth, but I was wondering if IMU sensor data is being used already or is there a way I can integrate the data in the current flow for better accuracy and mapping !

[SaurabhMishra2003]

Hi @matlabbe ,

I have one very important query. Actually I m planning on to integrate a detection model while mapping. So I m hoping for the structure to be like - I get frame -> I run my detection logic there -> send back these to native wrapper. But when I m using ARCore Java, I can access onDrawFrame, hence access to frame somehow. But when I m using ARCore NDK, the function is not entering into onDrawFrame. So, how will I get the frames here to run my detection algorithm ?

Thank you.

[matlabbe]

The IMU is already fused in the pose provided by ARCore.

Is your code running in Java or C++? For ARCore NDK, the frame is captured in c++ directly, so you won't have access from Java code. For both Java or c++ interface, these functions should be called with the frame.

If your code is in Java, you would have to stick with ARCore Java version.

[SaurabhMishra2003]

I am using ARCore NDK as Camera Driver with Depth from Motion enabled!

[matlabbe]

You may check if ARCore java can use a function like ArFrame_acquireDepthImage in c++, like here https://github.com/introlab/rtabmap/blob/49704743209316f13889a39a0ac617a5ad67aad5/app/android/jni/CameraARCore.cpp#L390

It seems it exists: https://developers.google.com/ar/reference/java/com/google/ar/core/Frame#acquireDepthImage() That could be added somewhere after that call: https://github.com/introlab/rtabmap/blob/49704743209316f13889a39a0ac617a5ad67aad5/app/android/src/com/introlab/rtabmap/ARCoreSharedCamera.java#L713 instead of the TOF camera approach.