Make godot.bindings code completion friendly

[prokopst]

trafficstars

godot.bindings is not code completion friendly due to use of lazy module binding.

For example while using PyCharm I just see import errors - godot.binding.Spatial in my case. It's not even possible to use dynamic runtime type inference, because PyCharm does not execute the code, Godot does.

Isn't there a better way to handle it? What does lazy loading trying to solve?

[touilleMan]

godot.bindings is indeed some black magic that is needed to lazy load the Godot API (e.g. Node2D, Engine etc.) in Python.

The good thing about lazy bindings:

• we don't have to track the Godot API changes because we get the binding informations at runtime using the GDnative reflexion feature.
• a non-lazy godot.bindings module would be an auto-generated mess full of classes declarations wrapping Godot API's classes. Loading time would be worst (but I would guess by not much) and more important file size would be much bigger.

On the other hand the bad things about lazy bindings:

• As you discovered we don't have support for autocompletion :'-(
• Code is more magic (as opposed to an "easily understandable ugly mess" with non-lazy bindings so not sure what is best...)
• We require the GDnative reflexion feature which is disabled in Godot release mode (currently we must use release-debug mode instead)

As you see there is pros and cons, the lazy-binding choice is a old one and could be no longer relevant now...

[prokopst]

we don't have to track the Godot API changes because we get the binding informations at runtime using the GDnative reflexion feature.

I wonder how often Godot API breaks backward compatibility. I'm a Godot newbie to be honest, so I have no idea. Ideally the API should be very stable, otherwise basically any similar project for a statically typed language would suffer. But hey, we are not in an ideal world.

a non-lazy godot.bindings module would be an auto-generated mess full of classes declarations wrapping Godot API's classes. Loading time would be worst (but I would guess by not much) and more important file size would be much bigger.

If I understand correctly when a module is imported it's evaluated just once. And since godot.binding is quite crucial to make anything happen - there is basically no laziness, because it will happen no matter what. But I don't know the internals of the project to be really sure.

I won't comment pros, because, you know, they are the pros 😄

[prokopst]

Btw, an alternative approach might be stub files: https://www.python.org/dev/peps/pep-0484/#stub-files

[touilleMan]

. And since godot.binding is quite crucial to make anything happen - there is basically no laziness

It's true that the default code template starts with a from godot.bindings import * which force the generation of all the binded classes. Obviously this is not mandatory and one could only import what he needs to optimize the loading time. However even with this big import the loading time is currently blazing fast so performance doesn't seem to be an issue here.

I wonder how often Godot API breaks backward compatibility.

Good question, back when the choice of lazy binding was made, there was no GDNative API so the only way to add 3rd party stuff was to create a new module and recompile Godot with it. So lazy binding was the only way to allow binding on those arbitrary 3rd party modules.

Afterward the GDnative API appeared but kept changing a lot so it was still a pita to track the changes.

Now that Godot 3.0 is released the Godot API should be guarantee to be backward compatible so using a static binding makes much more sens.

However this create the question of what to do to deal with backward compatibility:

• We can stick the binding with 3.0 API (but user wouldn't be able to use new API class/functions from Python)
• Or we could integrate API evolution after each Godot release, but what would happened if such a new feature is called from an older version of Godot ? (my guess is we should simply raise an exception but things can be more tricky in detail...)

tl;dr: Thanks for opening this issue, switching to static binding seems seems more and more like a good idea :-)

[prokopst]

These are good points. I guess they are the reasons why the C# binding is bundled with Godot and released separately.

From a user point of view, I don't mind using a separate Godot bundle to be honest. I'm playing with C#, so it's the usual way for me anyway.

BTW I see another benefit of the "non-lazy" binding, the code would not have any conversion logic for functionality returning specific types: https://github.com/touilleMan/godot-python/blob/e11fde2bc7e810e3194bc6ea6035ddd2dc87aa16/pythonscript/embedded/godot/hazmat/tools.py#L69 And the same for arguments: https://github.com/touilleMan/godot-python/blob/e11fde2bc7e810e3194bc6ea6035ddd2dc87aa16/pythonscript/embedded/godot/hazmat/tools.py#L187

[touilleMan]

BTW I see another benefit of the "non-lazy" binding, the code would not have any conversion logic for functionality returning specific types:

We would still need variant conversion for var args functions (which must be called with an array of variant as parameter) and for some other things like property getter/setter (in certain exotic cases)

Anyway, I've created the static-binding-generation branch. So far I've added a tools/generate_godot_bindings_module.py script that does static binding generation by reading as input Godot's doc xmls (btw @karroffel do you see any trouble doing this ? My hypothesis is godot/doc is automatically generated from the codebase so it should never be outdated or contain typos).

[prokopst]

So far I've added a tools/generate_godot_bindings_module.py script that does static binding generation by reading as input Godot's doc xmls

@touilleMan if it turns out to be a usable solution - nice out of the box thinking there. Is there any way I can contribute?

[karroffel]

@touilleMan I'm not sure, I would probably use the api.json that Godot outputs for generation, then use the XML files to add/lookup documentation. The json file contains some more information that the XML doesn't, mostly technical stuff, but if the python bindings don't need them then the XML files should be fine.

[touilleMan]

@karroffel Ho yeah ! I totally forget this api.json. However I don't see where it is generated (doing grep on the godot repo only gives me where gdnative_api.json is generated...)

@prokopst As you can see in my branch, the code for binding methods/members is missing in the template. The idea is to look into the pythonscript/embedded/godot/hazmat/lazy_bindings.py file and extract the code from there. This is a really tricky task however (even for me given I don't remember precisely how this code works anymore :laughing: ) If you don't fell comfortable with this, you can give a hand writing documentation (ideally should be rst in the docs folder, but even unformatted stuff is appreciated)

[karroffel]

@touilleMan ./godot.bin --gdnative-generate-json-api api.json will generate the json file. This only works with tool enabled builds, but it's only needed for class generation, so it should be fine.

[prokopst]

As you can see in my branch, the code for binding methods/members is missing in the template. The idea is to look into the pythonscript/embedded/godot/hazmat/lazy_bindings.py file and extract the code from there. This is a really tricky task however (even for me given I don't remember precisely how this code works anymore 😆 )

@touilleMan I'm slowly getting acquainted with the code and started with something very small - virtual methods. I'll prepare a pull request to your branch. But don't wait for me, just do whatever you want - I don't want to slow you down too much.

Btw I've noticed a trap already related to inheritance:

Python classes which represent Godot classes need to use inheritance to be able to access parents stuff. The problem is that Python's class definition is order sensitive, i.e. class B can't inherit from class A, when A is bellow B in the code. So we have to order classes first (build an inheritance tree?)

[cspotcode]

Python classes which represent Godot classes need to use inheritance to be able to access parents stuff. The problem is that Python's class definition is order sensitive, i.e. class B can't inherit from class A, when A is bellow B in the code. So we have to order classes first (build an inheritance tree?)

Python's from __future__ import annotations and if typing.TYPECHECKING: should avoid this issue. Since the annotations only exist for the language service, they never need to be executed. And ordering shouldn't matter to the language service.