[4.x] Add ARCore support

[paddy-exe]

trafficstars

This is an updated version for ARCore support of https://github.com/godotengine/godot/pull/47455 but for 4.x.

This is a WIP and feedback is very much appreciated

Procedure for building:

Generally follow this build instruction: https://github.com/GodotVR/godot_arcore

My 4.x branch of godot_arcore: https://github.com/paddy-exe/godot_arcore/tree/4.x

NDK version: 21.4.7075529 Android Studio Electric Eel (latest archived version)

Build the updated engine

1. Clone the repository with the necessary changes:

git clone --branch 4.x-arcore-support [email protected]:paddy-exe/godot.git

2. Build the engine

cd godot
scons platform=<PLATFORM>

3. Generate the extension files

cd bin
./godot.<PLATFORM>.editor.x86_64 --dump-extension-api --dump-gdextension-interface

Build the Plugin

1. Clone the repository of the plugin:

cd ../..
git clone --branch 4.x [email protected]:paddy-exe/godot_arcore.git

2. Download the godot-cpp submodule

cd godot_arcore
git submodule update --init --recursive

3. Copy the extension files to the godot-cpp submodule:

cd ..
cp godot/bin/extension_api.json godot/bin/gdextension_interface.h godot_arcore/plugin/libs/godot-cpp/gdextension/ 

4. Generate the binding classes

cd godot_arcore
python ./generate.py

5. Build the plugin

./gradlew :generatePluginBinary   

[BastiaanOlij]

Looking good so far, I wonder if the ARCore mode will still be needed now that we can append the needed manifest items through the ARCode AAR, but the camera server changes are definately needed.

[dsnopek]

Thanks for picking up this effort!

I only skimmed the code, but for the most part the changes seem fairly straight-forward! The only thing I don't understand is: what is the "ARCore" XR mode for?

My 4.x branch of godot_arcore: https://github.com/paddy-exe/godot_arcore/tree/4.x

Your branch appears to have vanished!

[paddy-exe]

Thanks for picking up this effort!

I only skimmed the code, but for the most part the changes seem fairly straight-forward!

Thanks for taking a look David!

The only thing I don't understand is: what is the "ARCore" XR mode for?

I am still new at the codebase but as far as I understand it myself Godot doesn't know which XR functionality you want to use (you can use OpenXR on Android as well for VR purposes) on export so that is why you have to tell Godot yourself by setting it in the export settings.

My 4.x branch of godot_arcore: https://github.com/paddy-exe/godot_arcore/tree/4.x

Your branch appears to have vanished!

Should be there again. I seem to have accidentally deleted it on the server-side. Had my local git backup luckily 🍀

Maybe you can also give your input here @dsnopek: In 3.x there was godot_android.h in GDNative. In 4.x the same functionality is in the JavaGodotWrapper class which is not exposed to GDExtension. We need this class to create an environment, get the devices activitiy and more. The question is how it should be done. @BastiaanOlij also mentioned that we could do it in Java.

[dsnopek]

Discussed at the GDExtension meeting: as far as we understand it, the methods and data that would be exposed would only be useful for GDExtension, and so it probably doesn't make sense to expose via ClassDB, where GDScript would have access to it. Instead, it might make more sense to add new functions to the GDExtension interface.

[m4gr3d]

I think it would be helpful to take a step back and clarify what we actually need in order to get ARCore working on Godot 4.x.

Maybe you can also give your input here @dsnopek: In 3.x there was godot_android.h in GDNative. In 4.x the same functionality is in the JavaGodotWrapper class which is not exposed to GDExtension. We need this class to create an environment, get the devices activitiy and more. The question is how it should be done. @BastiaanOlij also mentioned that we could do it in Java.

As @BastiaanOlij mentioned, most of the changes (asides from the camera related ones), and discussed functionality can instead be accessed through Godot Android plugin.

[BastiaanOlij]

As @BastiaanOlij mentioned, most of the changes (asides from the camera related ones), and discussed functionality can instead be accessed through Godot Android plugin.

I think the main issue @paddy-exe is currently running into is not having access to the JavaVM. Those were exposed to GDNative through that class. I don't think that is the way it should be done now, but the correct way of accessing this information is unclear.

I vaguely remember we did this somehow in early ports of the OpenXR plugin but I can't find any info on that anymore.

[paddy-exe]

@m4gr3d I am totally fine with doing it either way. I discussed a possibility to create a base class with @dsnopek and create an inheriting class for GDExtension access. I am totally fine with doing it either way. I discussed a possibility to create a base class with @dsnopek and create an inheriting class for GDExtension access. As Bastiaan already mentioned I have no clear understanding how to do this now with the current system.

[dsnopek]

@paddy-exe:

I discussed a possibility to create a base class with @dsnopek and create an inheriting class for GDExtension access. I am totally fine with doing it either way.

I'm not sure how the base class fits in anymore, after what we discussed at the GDExtension meeting. I had understood the base class idea to be a way to more cleanly expose JavaGodotWrapper via ClassDB. But as @vnen astutely brought up, this probably shouldn't be exposed to GDScript (which it would be if exposed via ClassDB), only GDExtension, which can be done by adding some new functions to gdextension_interface.h. In that case, you could maybe leave JavaGodotWrapper as it is, and then these new functions can just grab the data you need from it?

If you need help getting that going, feel free to ping me on RocketChat :-)

[paddy-exe]

I'm not sure how the base class fits in anymore, after what we discussed at the GDExtension meeting. I had understood the base class idea to be a way to more cleanly expose JavaGodotWrapper via ClassDB. But as @vnen astutely brought up, this probably shouldn't be exposed to GDScript (which it would be if exposed via ClassDB), only GDExtension, which can be done by adding some new functions to gdextension_interface.h. In that case, you could maybe leave JavaGodotWrapper as it is, and then these new functions can just grab the data you need from it?

Ah alright. Yeah the audio during the meeting wasn't great for me due to connection issues. I guess it would be quite hard to do this via the low level gdextension_interface.h.

If you need help getting that going, feel free to ping me on RocketChat :-)

I will revert the last commit and try to get into the Android ARCore plugin and expose the functions there like @m4gr3d suggested and if that fails will come back to your offer. Thanks!

[paddy-exe]

Alright, reverted back to original commit and supplied the updated CameraFeed.xml so CLI builds successfully without errors.

[Colbix]

Is there any further progress for the ARCore plugin? It looks like you were making headway and only pending a plugin finalize from the 4.x refactoring.

[paddy-exe]

Is there any further progress for the ARCore plugin? It looks like you were making headway and only pending a plugin finalize from the 4.x refactoring.

I am still interested in getting this going. I was finalising my thesis and had no time until now. I will try to dedicate some time to this next year hopefully👍🏻

[BastiaanOlij]

Looking at this during our ARCore meeting, this will likely be able to be closed, camera feed logic is being implemented in #95187, display rotation can now be accessed directly in the plugin and no longer needs changes in Godot, but we're unsure about the ARCore mode.

@m4gr3d do you know if the ARCode xr-mode would still be needed or that what Luca does here is enough?

[dsnopek]

Is anything in this PR still needed anymore? Or, is it superseded by https://github.com/godotengine/godot/pull/96705?

[paddy-exe]

Is anything in this PR still needed anymore? Or, is it superseded by #96705?

I don't think so. Will close the PR then