[Feature]: facilitate custom protoc implementations, improve surrounding interfaces

[ascopes]

Brief description

As part of Quarkus gRPC Zero, @carlesarnal, @roastedroot, and @andreaTP have started a project at https://github.com/roastedroot/protobuf4j that aims to provide a WASM-ified build of protoc that can run within the JVM runtime.

This project enables building protobuf sources in a platform-independent way, by distributing a runnable JAR (or equivalent) that invokes the WASM that has been transpiled from Google's C/C++ sources (with some patches to handle platform specific details that would otherwise be incompatible with the WASM runtime.

This issue is to facilitate being able to integrate Protobuf Maven Plugin with that solution, as well as any other potential projects or forks in the future.

Goals

• Allow optional overriding the group ID, artifact ID, classifier, and type of the protoc artifact.
• Continue to allow the string based specification in current versions (i.e. <protoc>4.33.1</protoc>, <protoc>PATH</protoc>, and <protoc>https://...</protoc>). We can consider deprecating this behaviour with a warning but it must be kept until the next major version.
• Handle cases where protoc artifacts are JARs rather than native executables. This specific functionality can come at a later date as the second part of preparation for integration with the protobuf4j build artifacts if that is more appropriate. The rest of this scaffolding can be implemented prior to this as it also facilitates being a long-overdue improvement.

Other places we can improve on as part of this work

• Add the ability to override the artifact that protoc is pulled from (e.g. to customize the classifier, change group/artifact IDs, etc). This should be able to be supported as a side effect of this work.

Considerations, questions, things to do

The following is a list of things to do, things to investigate, questions to answer, strategies to assess.

• [ ] Allow optional overriding the group ID, artifact ID, classifier, and type of the protoc artifact.
  • [ ] A custom Plexus Converter will likely be needed to allow the existing bare-string parameters to still work correctly.
  • [ ] This should allow specifying an empty classifier, which may require additional work since we currently infer the classifier from the current platform per the naming of protoc artifacts on Maven Central.
• [ ] We most likely will need to be able to flag to this Maven plugin whether to treat the protoc distribution as an executable or a JAR.
  • [ ] Can this just check for a ZIP header on the artifact to verify if it is a JAR or not?
  • [ ] Will the above complicate any more obscure use cases in the future (such as running Python JARs)?
  • [ ] Should we be facilitating shebang detection on JARs for both this and Protoc Plugin management?
• [ ] Change the protoc resolution logic for Maven dependencies to support dependency resolution depth and to default to a deep resolve of all transitive dependencies.
  • This is needed so that we can correctly resolve dependencies of the protoc WASM build.
• [ ] Add the ability to differentiate between pure binaries for protoc and Java executables that should be invoked within a JVM.
  • Do we want to load JVM artifacts into a ClassWorlds realm to run them within the same JVM, or should we prefer forking a new JVM like we do for protoc plugins?
  • [ ] ...if the former, investigate how this would work?
    • This would remove the need to centralise the JVM plugin resolution logic for bootstrapping executables from JARs.
    • JVM flags would be fed in from Maven.
    • Likely much faster as we may be able to cache the invocation's classes in memory after the first build, resulting in much faster multi-goal and multi-module builds. Potentially much faster with mvnd as well.
    • [ ] Is there a material risk of toppling the JVM running Maven, or is this unlikely to be problematic?
  • [ ] ...if the latter, refactor the JvmPluginResolver to move out all logic around constructing an executable Java entrypoint so that it can be reused here. This will also facilitate using this Maven Plugin with the sources of Protobuf4J directly before they are squashed into a JAR, which will allow for debugging any issues locally.
  • [ ] Consider adding the ability to attach debuggers to JVM protoc plugins and the new protoc mechanism, or documenting instructions for configuring the JDWP server.
  • [ ] How will passing JVM arguments (e.g. -Xms128m -Xmx512m -XX:+UseZGC) work?
  • [ ] Should we provide sensible JVM flag defaults as part of this like we do for JVM protoc plugins?
• [ ] Prepare Plexus integrations

  • [ ] Since we need to support passing objects as part of <protoc/> we may want to condider if implementing some generic shim around plexus converters for sealed interfaces/classes is useful. This would also enable consolidating <binaryMavenPlugins>, <jvmMavenPlugins>, <binaryUrlPlugins>, <binaryPathPlugins> into a single parameter that can use an attribute as the discriminator, e.g.

    <protocPlugins>
      <protocPlugin kind="maven-java">
        ...
      </protocPlugin>
      <protocPlugin kind="url">
        ...
      </protocPlugin>
      <protocPlugin kind="maven-binary">
        ...
      </protocPlugin>
    </protocPlugins>
    

  • [x] Move plexus components to a centralised package for consistency. -- See PR GH-878.

Further details

This will be a multistep process and is expected to span across multiple MRs and possibly multiple releases.

Would you like to contribute to this feature?

• [ ] Yes, I'd like to contribute this change myself!