Kotlin reflect doesn't work

[Humberd]

Let's say I want to check if a class has an annotation:

packet::class.hasAnnotation<HotPacket>()

In order to achieve that I populated build.gradle.kts with

implementation(kotlin("reflect"))

Unfortunately I get this error:

kotlin.jvm.KotlinReflectionNotSupportedError: Kotlin reflection implementation is not found at runtime. Make sure you have kotlin-reflect.jar in the classpath

[raniejade]

This is due to our gradle plugin explicitly ignoring kotlin dependencies since we ship a specific version together with the binary.

Might be good to have a includeKotlinReflect in the gradle plugin setting. We currently treat kotlin as a platform, so you can't customize the version that you want to use - the specific build itself support must support it.

[Humberd]

What is the reason for shipping a specific version of Kotlin?

[chippmann]

The reason is that we need a shadowJar (fat jar) with all dependencies in it. Both for the godot-bootstrap.jar and the main.jar As godot-bootstrap.jar is loaded first (as it handles loading and unloading of the main.jar), the kotlin dependencies have to be in godot-bootstrap.jar. At the time if building that, we don't know yet what kotlin version the user want's to use. But we need all kotlin dependencies otherwise it cannot run. As your code resides in main.jar which is then loaded by godot-bootstrap.jar we cannot have kotlin dependencies in it, as otherwise all classes would be present twice. Hence we atm just exclude kotlin dependencies when assembling the main.jar. But yeah a case like yours is exactly one we don't handle atm.

But I'm unsure how we should handle that. Probably we need to check which dependencies are already in the godot-bootstrap.jarwhen assembling the main.jar and only exclude them, rather than just exclude all kotlin dependencies with a wildcard.

[chippmann]

But that of course doesn't handle the kotlin version case. For now we decided that we just support the kotlin version with which we build. As we need to ship the godot-bootstrap.jar alongside the engine. We should probably mention that in the docs as well. That said, if you have an advanced usecase where you need a special kotlin version, you could just manually copy the godot-bootstrap.jar from build/libs to the editor executable root an override the one we ship with the engine. Probably also something that could go in the Advanced section of the docs

[Humberd]

So If I understand correctly the godot-bootstrap.jar shipped with godot has already included predefined Kotlin version. On the other hand when I build the game another godot-bootstrap.jar is being built, which resides in build/libs. This means I can move the newly built jar next to the godot.exe and this way I could include reflections?

[chippmann]

No not reflections, just a different kotlin version. The dependency exclusion is still a problem.

[chippmann]

Hmm wait. I was telling nonsense. This is only the case for local builds not shipped ones. So yeah currently the kotlin version is fixed and kotlin dependencies are always excluded. The latter we can fix. The former can only be circumvented if you build the project locally.

[raniejade]

As godot-bootstrap.jar is loaded first (as it handles loading and unloading of the main.jar), the kotlin dependencies have to be in godot-bootstrap.jar.

Initially this was part of the projects build however, the editor may crash if the project isn't built yet. @chippmann is there a way we can delay initialization of the module only when godot-bootstrap is present (reload when necessary if it changes)? I vaguely remember we talked about some edge cases when reloading.

[chippmann]

I don't know if we can delay the module initialization. I think not. But what we tried is to reload the whole bootstrap jar which failed because of classloader issues. We tried that pretty long IIRC and settled with just shipping the bootstrap jar. The root problem is IIRC that we need the same classloader for loading different jars. And to have a classloader, we need the kotlin dependencies. So even if we would abstract it further by having not the bootstrap shipped, just a "runtime" or something, it would still need to have the kotlin dependencies in it. Leaving us with the same problem again. Also we cannot just destroy the jvm and rebuild it because of the limitation that a process can only spawn one embedded jvm in it's lifetime.

[raniejade]

I don't know if we can delay the module initialization

Sorry, I didn't mean the module itself but our bootstrap logic - i.e creating the classloader that loads the bootstrap.jar.

[Sipaha]

KotlinModule for jackson ObjectMapper also doesn't work without reflection lib:

dependencies {
    implementation(group = "com.fasterxml.jackson.core", name = "jackson-databind", version = "2.12.3")
    implementation(group = "com.fasterxml.jackson.core", name = "jackson-annotations", version = "2.12.3")
    implementation(group = "com.fasterxml.jackson.module", name = "jackson-module-kotlin", version = "2.12.3")
}

val mapper = ObjectMapper()
try {
	mapper.registerModule(KotlinModule())
} catch (e: Throwable) {
	GD.printErr("Error ${e::class} msg: ${e.message}")
}

Error class java.lang.NoClassDefFoundError (Kotlin reflection is not available) msg: kotlin/reflect/jvm/internal/KotlinReflectionInternalError

Without custom try/catch for Throwable (NoClassDefFoundError is not an Exception) this error displayed in logs in this way: without information about source of problem.

[chippmann]

I did some testing on this subject and the problem is either very trivial to solve or pretty hard to solve: Given we ignore the kotlin version constraint entirely (we need this constraint anyways because of the compiler plugin).

If i remove the following line inside our gradle plugin: it.exclude(it.dependency("org.jetbrains.kotlin:kotlin-stdlib.*")) it does not make a difference. The reflect dependency has to be applied to the godot-bootstrap. If it's only applied to main it has no effect. My guess is that this has to do with the fact that we load main with the classloader from within godot-bootstrap and the reflect dependency has to be there as reflection is probably happening there.

So if i add the reflect dependency to godot-bootstrap it works. So I added a configuration for our gradle plugin with which one can add dependencies to godot-bootstrap which looks like this:

godot {
    bootstrapDependency("org.jetbrains.kotlin:kotlin-reflect:${kotlin.coreLibrariesVersion}")
}

With it the defined dependencies are added to the godot-bootstrap. This works. But only when the game/application is started from the commandline or was exported. Not if it is run from within the editor. The reason being the selection we make from where to load the godot-bootstrap.jar:

#ifdef TOOLS_ENABLED
    String bootstrap_jar{OS::get_singleton()->get_executable_path().get_base_dir() + "/godot-bootstrap.jar"};
#else
    String bootstrap_jar{ProjectSettings::get_singleton()->globalize_path(vformat("user://%s", bootstrap_jar_file))};
#endif

The reason we have this selection is that we cannot just load the godot-bootstrap.jar from the users build dir, as when a clean happens, the jar is just deleted even if it's in use by the editor. We can also not just reload it because of the issues mentioned earlier in this discussion.

So how should we proceed? I see the following options:

1. So far we only encounter this issue for the reflect dependency and it's unlikely that another dependency has the same constraint. So we could just add the reflect dependency directly like we do with the stdlib so its present anyways. Problem with this: The godot-bootstrap.jar's size increases from 5.6MB to 8.8MB just from this dependency alone. Which is not ideal.
2. We make it configurable what dependencies should be added to godot-bootstrap.jar (see my example above). This means it'll work for games/applications started from the commandline (or when exported) by default. But means that the editor has to be manually restarted each time the godot-bootstrap.jar has changed. Also to prevent crashes and errors during clean, we would need a commandline arg, with which the user can specify a custom location for the godot-bootstrap.jar. Obviously the downside of this variant is the manual labor involved: Passing a commandline arg to the editor with a custom location, manually copying changed bootstrap jars and manually restarting the editor.

Sadly i see no other options. Or do i miss one? IMO we should go with option 1. It's a one line change, and i guess the size increase is justifyable. And if one really needs those few MB's, he can just build the engine himself with the reflect lib removed.

@piiertho , @CedNaru and @raniejade What's your take on the subject?

[piiertho]

@chippmann Maybe we can include dependency in the godot-bootstrap.jar shipped with editor, and make it optional for the one which is built by user.
This way, we can still work in editor, and configure it for exports. The only problem I see with this is that editor jar and "exported" jars could be different, which can lead to hard debugging.

[raniejade]

I say just include it always. Then maybe later, we could include an option to exclude kotlin-reflect during export.

[AutonomicPerfectionist]

Has there been any progress on this issue? If not, how would one go about manually adding reflect to the bootstrap?

[chippmann]

@AutonomicPerfectionist no not yet. Neither of us got around to do the necessary changes yet. I might have a look tomorrow. If not: adding the reflect dependency here should be enough: https://github.com/utopia-rise/godot-kotlin-jvm/blob/master/kt/plugins/godot-gradle-plugin/src/main/kotlin/godot/gradle/projectExt/setupConfigurationsAndCompilations.kt#L34 You then need to copy the resulting bootstrap.jar alongside the editor executable and thus override the one we ship. For exported builds it should then work without copying anything.

[piiertho]

I wonder if runtime crash we had when testing with graalvm native-image on iOS is not linked to this issue.

We had Unreacheable code when calling ::class, whatever the class was.

[chippmann]

I wonder if runtime crash we had when testing with graalvm native-image on iOS is not linked to this issue.

We had Unreacheable code when calling ::class, whatever the class was.

I think more that it was the issue we discovered in our last meeting: that we did not set the classloader and contextClassloader in the Bootstrap.kt class when doing Bootstrap::doInitGraal like we do in Bootstrap::doInit