Consider embracing C# Source Generators?

[jonpryor]

C# Source Generators have been announced. They aren't a C# language feature; they're not LISP-style macros. Instead, a class implements the Microsoft.CodeAnalysis.ISourceGenerator interface, and the ISourceGenerator.Execute(SourceGeneratorContext) method is executed as part of the compiler "pipeline". SourceGeneratorContext provides access the the C# source code parse trees that exist at the time of ISourceGenerator.Execute() invocation, and can emit new C# source code to be included into the final executable.

(@jonpryor has no idea if ISourceGenerator.Execute(SourceGeneratorContext) can be invoked multiple times at different times of the parse tree, so let's assume that it's only executed once.)

Is this useful for Java binding?

Replace generator?

One can imagine a world in which generator.exe becomes a C# source generator, and emits bindings as part of the C# compiler invocation process.

Other than being "cool", @jonpryor isn't sure if this actually buys us anything else useful. (It would still be cool!)

@jonpryor can't, offhand, see any ways to significantly alter the existing binding process; there would still be a Metadata file, an api.xml, etc.

Replace jnimarshalmethod-gen?

The C# source generator announcement suggests that C# source generators may be useful for Ahead Of Time (AOT) compilation purposes, which is a similar area as what jnimarshalmethod-gen.exe targets. One can imagine all Xamarin.Android projects as having a "post-processing" source generator which looks at all Java.Lang.Object and Java.Lang.Throwable subclasses, then emits a __<$>_jni_marshal_methods-style nested class which contains marshal methods for all method overrides, and a __<$>_jni_marshal_methods.__RegisterNativeMembers() which performs the actual runtime registration. This would allow everything to happen at compile time.

Another upside is that, at present, jnimarshalmethod-gen.exe cannot run under .NET 5, as it uses AppDomains; see Issue #616. This would circumvent that limitation.

The downside to using source generators in this manner is twofold:

1. All "Java callable" types which override Java methods would need to be declared as partial classes.
2. It hardcodes the marshal method ABI, and
3. Interaction with custom marshalers is unknown.

(2) is a somewhat "silly" complaint: the marshal method ABI is currently hardcoded, by nature of the interaction of the RegisterAttribute.Connector property, which (eventually at runtime) invokes a corresponding n_*() method -- also contained in the binding assembly -- which performs the Java-to-managed parameter & return type marshaling.

@jonpryor still wants to remove the need for RegisterAttribute.Connector; see the Proposed Java.Interop Architecture section.

Then there's the idea of C# Function Pointers (see also 8e5310be809477e03cd6c4ed4137c903dfe9991c).

In an architectural view in which jnimarshalmethod-gen.exe runs at app build time, and has access to all of the assemblies that will be present in the final app, RegisterAttribute.Connector becomes unnecessary, and jnimarshalmethod-gen.exe could migrate to a C# Function Pointer-compatible IL implementation in a way that doesn't require rebuilding all existing binding assemblies. This is a cool feature!

Custom marshalers (3) (da5d1b8103bb90f156b93ebac9caa16cfc85764e) are another "not quit working" idea that would be impacted by embracing source generators. The current architecture is that Java.Interop custom marshalers work by using [JniValueMarshalerAttribute] and a JniValueMarshaler<T> subclass: the JniValueMarshaler is created at jnimarshalmethod-gen.exe execution time to create System.Linq.Expressions.Expressions to deal with marshaling Java parameters to managed parameters, and managed parameters to Java parameters.

(3.a): Is there a way to get C# source generators to "play nicely" with Expressions? Probably not, though this hasn't been explored. If there's no way to make this work, then it wouldn't be possible to support custom marshalers alongside source generators.

(3.b): if there is a way to get C# source generators to play nicely with expressions, we then get a "versioning" problem: the marshal method generated would be based on the custom marshaler available at compilation time, which may be a different assembly version than is bundled with the resulting app. This is kinda/sorta like the difference between C# const and readonly: const values are "baked into" the call-site, while readonly values are actually looked at runtime. Custom marshalers would be in a similar position: the emitted marshaling code would be "baked into" the referencing assembly at assembly compilation time, which is not app build time. If (when) the assembly containing custom marshalers changes, those changes won't be "percolated" ..

At present, C# source generators don't appear to be a good idea for fully replacing jnimarshalmethod-gen.exe.

[jpobst]

I've thought about if Source Generators are useful in a generator context, and I think the biggest issue I see is possibly increased compile time. Today we are able to generate the source code and we only have to re-generate it when the jar or metadata changes.

In a world where Roslyn runs us every time the compiler is running, I'm not sure we can know whether we need to re-generate the entire binding. This gets worse in a .NET5+ world where there isn't a distinction between binding projects and applications, and you may have added the .jar directly to your application project. In this scenario we may be triggering expensive regeneration with every change the user makes to their application.

One benefit, though it may be tricky, would be that a Source Generator generator could be aware of other C# code in your project, like in an Addition. Today you cannot define an enum as an Addition and then use metadata to change a bound method to use that enum, because generator isn't aware of any C# you have written for the binding. generator will see the type as missing and will not bind the method because it cannot resolve the type.