Allow creating GDExtension plugins from inside the Godot editor

[aaronfranke]

This PR implements https://github.com/godotengine/godot-proposals/issues/3367 (not exactly as that proposal describes, but solves the same problem).

The status quo for GDExtension development is really unfortunate. For the most part the only easy way to get started is to copy-paste an existing GDExtension project and use it as a template. If you don't do this, you'd need to follow the documentation, which is a massive behemoth that manually explains to the user how to create each file from scratch, and it would take a user hours to follow this guide.

Really, I believe it should be as simple as making a GDScript plugin:

• Click a button to open a wizard to make a plugin.
• Give your plugin a name.
• Select GDExtension from the dropdown menu (only step added compared to GDScript plugins).
• Click Create.

This is what this PR implements. Just go into Project Settings -> Plugins, click Create New Plugin, and select GDExtension.

As mentioned by proposal https://github.com/godotengine/godot-proposals/issues/3367, this sets up the build system with a build command (scons), the .gitignore files, the .gdextension file, the type registration boilerplate, and it even git clones godot-cpp. By default, it will also initiate scons to compile the extension, but this does take awhile, so you can skip this step by unchecking the "Compile now? (slow)" box.

Note that this PR does not include setting up system-level dependencies, such as installing a C++ compiler, installing Git, installing SCons, etc. This varies widely by operating system and it would be a slippery slope to try and have that. This PR is purely focused on the files that go in your project, not the system-wide dependencies. This PR does not try to detect a C++ compiler, but it will warn the user if either Git or SCons is not found on their system.

When you create a GDExtension plugin with this PR, one example node is provided called ExampleNode. Like the Blender default cube, this is expected to be replaced by users as soon as possible. However, its existence is useful to both new and experienced users, since it serves 2 functions: 1) As a reference for how to make a node type, including the code, registration, docs, icon, etc, and 2) A minimal test to check if your GDExtension plugin works (don't see it in the editor? It's not working).

Assuming you have the system-wide dependencies installed, and you leave the "Compile now?" checkbox enabled, and you wait for it to finish, the extension immediately works. No fuss, the example node is just there:

If you change the code, you just need to run scons in the root. A future idea would be to detect this in projects and provide a "Build" button in the editor UI like we already do for C#, but that is technically separate from what this PR does.

The files look like this:

The SConstruct at the root is extremely barebones, it is only one line to call into another SConstruct. It is designed to be simple enough so that more GDExtension plugins can be added to the same Godot project with no manual work. The _ensure_file_contains helper function in this PR completely handles the merging of more calls into this file. Except for deleting the first plugin's ExampleNode, you can pretty much just make 2 plugins back-to-back with no conflicts. If you want to have 20 GDExtension plugins, this PR will let you do that just fine, with one call to scons in the root folder able to compile all of them at once.

This PR provides 2 options for making a GDExtension plugin: "GDExtension C++ only" or "GDExtension C++ and engine module". The former creates a standalone GDExtension C++ plugin, with the only files stored in the root of the project being a .gitignore and barebones SConstruct.

The engine module option provides a few more files like SCsub and config.py, provides lots of #ifdef directives to support both GDExtensions and modules, and places most of the C++ code in the root of the folder, which is more suitable for making a Godot engine module. This way if users want to target both GDExtension and modules with one codebase, they can do this as easily as selecting it from the dropdown. This option turns the project's root folder into a module, so only one module is allowed per project:

When in this mode, you can either compile for GDExtension the same as with the "only" option, or you can copy-paste the entire Godot project into the engine modules folder, naming the folder the same as the addon folder name, and it works.

The source files are designed to handle the extension+module case, but are automatically simplified by this PR's project generator when making a non-module extension, including by looking at the #if preprocessor directives to determine which lines of code to exclude.

TODO list before undrafting this PR:

• Merge PR #90931
• Merge PR #90975
• Add support for documentation, see PR https://github.com/godotengine/godot/pull/83747 and proposal https://github.com/godotengine/godot-proposals/issues/8245
• Decide how we want this to work for non-C++ languages (currently the code is not designed to be plugable).
• Decide if we want this to work for non-SCons build systems.
• Get lots of feedback and testing. I'm sure this can be improved, but as-is it is already a massive improvement.

[Riteo]

Wow, this looks great! I wonder, does this also resolve https://github.com/godotengine/godot-proposals/issues/9306 ? It looks like the same problem IMO.

[Naros]

This looks great, but I also definitely think its worthwhile to consider support for other build systems beyond scons. There has been a tremendous amount of work put into the cmake build in godot-cpp, and I think its at least worthwhile for supporting that build system if someone picks "GDExtension C++ only".

[Lazy-Rabbit-2001]

This looks wonderful as I've been annoying with handling the preparation of making GDExtension manually for days. However, I think there could be an option to decide if godot-cpp is global or not, as some devs may use the same godot-cpp to avoid repeated compilations on this folder because of their parallel development on multiple GDExtensions. For example, according to current status of pr, a dev may produce two GDExtensions like this:

addons

exm_a

src

godot_cpp

exm_b

src

godot_cpp

See? There are two godot_cpp-s in seperate projects, but the dev may hope to use only one for these two GDExtensions, like this:

addons

_bin

godot_cpp

exm_a

src

...

exm_b

src

...

As the direct placement of folder godot_cpp may bring ambiguity and vagueness to devs, it's better to place it in a single common folder named _bin or something else clear that resonates with the devs.

[aaronfranke]

@Naros The big question I have is what would this feature look like? Should cmake be included as one of the supported options out of the box? Should it be plugable? If so how? I guess plugins should add more options to the dropdown menu, so we would have "GDExtension C++ only (scons)" and "GDExtension C++ only (cmake)"? Or is it worth refactoring this menu to have a more generalized "GDExtension" option and a second dropdown for "C++ SCons", "C++ CMake", etc?

@Lazy-Rabbit-2001 I agree, but there are some technical challenges when trying to make the plugins share godot-cpp. When I try to do this, and both call into the same SConstruct, it is unable to compile from the top-level SConstruct:

scons: warning: Two different environments were specified for target gen/src/variant/utility_functions.cpp,
        but they appear to have the same action: scons_generate_bindings(target, source, env)
File "/Users/aaronfranke/workspace/projects/gdextension-test/addons/godot-cpp/tools/godotcpp.py", line 463, in _godot_cpp

scons: *** Two environments with different actions were specified for the same target: src/godot.macos.template_debug.universal.o

I would need some help from the build system gurus to make this work.

[dsnopek]

Thanks for taking this on! It looks really cool :-)

I did a quick test of the PR, and the "Compile on" checkbox didn't seem to work for me. It detected scons and git, but it encountered an error when trying to run it:

scons: Reading SConscript files ...
Auto-detected 32 CPU cores available for build parallelism. Using 31 cores by default. You can override it with the -j argument.
Building for architecture x86_64 on platform linux
KeyError: 'arch_suffix':
  File "/home/dsnopek/Sync/Projects/games/gdext-creation-test/SConstruct", line 4:
    SConscript("addons/zap/SConstruct")
  File "/usr/lib/python3/dist-packages/SCons/Script/SConscript.py", line 661:
    return method(*args, **kw)
  File "/usr/lib/python3/dist-packages/SCons/Script/SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "/usr/lib/python3/dist-packages/SCons/Script/SConscript.py", line 287:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "/home/dsnopek/Sync/Projects/games/gdext-creation-test/addons/zap/SConstruct", line 35:
    env["arch_suffix"],
  File "/usr/lib/python3/dist-packages/SCons/Environment.py", line 405:
    return self._dict[key]

If I change the SConstruct to remove the env["arch_suffix"] part, then it builds fine when running scons manually. After that, the ExampleNode class appears and I can add it to my scene!

It looks like this creates a plugin.cfg file, which leads to an entry on the "Plugins" tab in "Project Settings". However, GDExtensions aren't plugins in this sense, and enabling/disabling the created plugin there will actually have no effect. We do plan to eventually add the ability to enable/disable GDExtensions in "Project Settings", but work hasn't started on that yet.

However, GDExtensions are quite different from plugins in many ways. It may make sense for GDExtensions to have their own tab distinct from "Plugins"? And, if so, then the UI here should probably be disentangled from the "Create New Plugin" form and process.

Anyway, this is a great start!

[Naros]

The big question I have is what would this feature look like? Should cmake be included as one of the supported options out of the box? Should it be plugable? If so how? I guess plugins should add more options to the dropdown menu, so we would have "GDExtension C++ only (scons)" and "GDExtension C++ only (cmake)"? Or is it worth refactoring this menu to have a more generalized "GDExtension" option and a second dropdown for "C++ SCons", "C++ CMake", etc?

So first, I think David is right, we should completely separate GDExtension from GDScript plug-ins here, 💯 . An independent UI has several benefits:

• Avoids confusing GDScript plug-in creators with C++ / GDExtension noise.
• Allows creating a screen tailored for GDExtension and its toolings.

On the GDExtension create dialog, I could imagine this being a bit similar to Project Settings with a tree on the left, and a panel on the right where we display context-specific options based on the choice on the left. The left panel tree might look like this:

- General 
- Build, Execution, and Deployment
  - Toolchains
    - Custom compiler
  - CMake
  - Meson
  - Scons
  - ...
- Plugins
  - ... 

The General tab allows you to define basic data about the extension, its name, a custom entry point name, the base path for where the compiled binaries should be copied, i.e. res:\\addons\<addon-name>\bin being the default. Additionally, this is where the user can define the minimum compatibility, hot-reload, etc.

Under the Build tab, we'd have sections for each supported build system that would be output for the user. These tabs could include specific options that can be customized for that specific build system. For example, here's what I could imagine being the case for CMAKE:

The Plugins section is really some future scope I would say, but could be where the build files get customized, additional classes added, or some other customizations are done that can be tailored to an organization's needs.

---

And circling back to Limbo's idea of a shared godot-cpp, this got me thinking about whether the C++ code should really reside in the project folder itself or would it make sense to reside in something like user://, with top-level paths like:

• user://godot-cpp (shared godot-cpp)
• user://gdextension/<extension-name>

I'm just not sure I like the idea of embedding the C++ code into the Godot project, but I'm not 100% sold either way. But now I do question if some of these settings belong configurable in the Project Settings and not in the "Create Extension" dialog 🤔