KhronosGroup
Prepare Java release to Maven Central
Addresses https://github.com/KhronosGroup/KTX-Software/issues/624
There is already some discussion about the related tasks and possible approaches in that issue, mainly starting at this comment. I'll try to summarize the current state of this PR and the open questions.
Organizational
One question that is not (yet) addressed with the PR is that of the actual deployment of the JARs to Maven Central. I'm not deeply familiar with (or may have forgotten) some of the details here. Maven is incredibly powerful, and it is possible to handle much of that release with Maven itself. In a perfect world (i.e. one that only involves Java 🙂 ), it is possible to set up Maven in a way so that a single Maven run can 0. define the version number for the release, 1. create a Git tag for this release, 2. check out this tag, 3. perform the build, 4. run the tests, 5. sign the JARs, 6. upload the JARs to the Maven Central 'staging' repository, 7. update the version numbers, ... (and more... heck, it can do anything, for what it's worth...)
However, for libraries that involve native libraries (as it is the case here), there are apparent caveats. For example, the run to build Linux- and Windows binaries can hardly happen on the same machine. (In theory, maybe, yes, but ... now throw in MacOS, and you're screwed...).
One reasonably painless path that I've been taking (for JCuda and JOCL) is:
- Build the native libraries for Windows, Linux, and MacOS, somewhere and somehow
- Throw these natives into a common directory
- During the Maven build: Copy these natives into the target directory, to include them in the JAR files
- Semi-manually sign these JARs
- Upload these JAR files to Maven, manually, via Sonatype, "Bundle upload"
Maybe someone wants to chime in with better approaches. (Preferably, going beyond the comment "You should use a better approach" ... ). In any case - regardless of whether it's fully automatic or semi-manual - there are some questions about authorization and authentification (for signing and the actual upload) where some Khronos Webmaster/Maven-Admin will have to contribute.
The current main state of the pom.xml contains a few fragments of the infrastructure that are currently <!-- commented out --> . Maybe some of that can be re-used?
Technical
The current state of this PR contains a few building blocks for the technical part of the deployment.
I added a copy of the LibUtils that I've also been using for JOCL. It is a class that can load native libraries, depending on the target OS and architecture, either from local files, or from a JAR file. In order to do that, it unpacks the files from the JAR into a "temp" directory, and loads them from there. This class can be used to conveniently ensure that the required native libraries are loaded when any of the classes is loaded that uses native functions.
I updated the Maven pom.xml to prepare the handling of the native libraries:
(One detail: I lowered the Java version requirement to 1.8. There is no reason to require Java 11 here...)
I added an execution of the maven-resources-plugin that takes all native libraries that are found in a /nativeLibraries subdirectory of the project, and puts them into the target/lib directory. From there, they will be packed into the JAR. (The libraries should be put into the nativeLibraries directory by the native build processes)
Right now, this is tailored to creating a single JAR that contains all native libraries. (This is the approach that I took for JOCL). Having "one JAR to rule them all" is convenient (e.g. for deployment). But it has drawbacks:
- The resulting JAR may be huge (depending on the size of the native libraries)
- It's not easy to create that JAR on one system (because it needs all the native libraries at once)
A better approach usually is to create dedicated JARs for each of the native libraries. This basically means that there would be
ktx-0.0.0.jarfor the main Java partktx-natives-0.0.0-windows-x86_64.jarcontaining only the Windows DLLktx-natives-0.0.0-linux-x86_64.jarcontaining only the Linux natives...- ...
This does involve some overhead for the POM: The right dependencies have to be added using profiles, but I already did that for JCuda (e.g. in the JCuda parent POM), and the same approach is used for other native libraries (like the LWJGL family of libraries, or java-cpp).
The next steps here will be to prepare the build infrastructure for creating the dedicated ktx-natives... libraries where each contains only the natives that are required for the respective OS/architecture.
One point about the naming of the native libraries that may look like a detail, but is pretty important:
The native libraries that are supposed to be packed into the JAR will need a version number suffix, like ktx-4.3.2.dll.
The reason:
When loading a native library in Java with System.loadLibrary("example"), then it will try to load the library (like example.dll) from a variety of possible sources, including the system PATH (environment variable). When someone has installed KTX-Software, then the directory that contains the ktx.dll and ktx-jni.dll will be added to the PATH. So when someone has installed KTX-Software 4.2.0, but uses the Java JAR for version 4.3.0, then the Java bindings would load that "old" ktx-jni.dll from the PATH, and may try to call a function that simply didn't exist in 4.2.0 - causing a crash...
A few things to note:
- CI is set up so when a release tag is added (e.g. v4.3.2) is added to the repo, it builds and deploys a release with that name. At present the Java part of that deployment consists of copying packages to GitHub releases.
- The Java build is taking the version number from the CMake variable that is set by the tag. It is important that we do not have to set tags or version numbers anywhere else.
- The native libraries already are named with full version numbers and said name is available in a CMake variable,
LIBKTX_FULL_VERSION, I think. This is standard practice for shared/dynamic libraries. The common name used by developers is a link to the file with the full name. - We handle the multiple native builds problem for Python. We have a GitHub Action that is manually run after all builds for a release have completed successfully. It downloads the native libs from GitHub Releases, assembles the packages to upload to PyPI and then uploads them. See
.github/workflows/publish-pyktx.yml. We can do something similar for Java. This cannot be automated at present because there is no way to programmatically determine that all builds have completed then rigger this action. If all builds are moved to GitHub Actions it may become possible.
@MarkCallow All that sounds OK to me. It sounds like there could be a path to properly handle the versioning, naming of the native libs, and (if the Travis->GitHub transition progresses) the packaging of the natives into JARs. Ideally, this could include the upload of the JARs to Sonatype/MavenCentral (maybe not strictly required, but certainly desirable).
However, there already are places that carry a version number that would have to be adjusted manually. Specifically, the main pom.xml currently declares the Java version of the KTX libraries to be 4.0.0-SNAPSHOT at https://github.com/KhronosGroup/KTX-Software/blob/3038f7cae9c875c0060ebb3d809e031ce088707f/interface/java_binding/pom.xml#L11 . I preliminarily and manually changed this to 4.3.3-SNAPSHOT in this branch, but we'd have to check where this version will come from when all this is to be automated.
However, there already are places that carry a version number that would have to be adjusted manually. Specifically, the main
pom.xmlcurrently declares the Java version of the KTX libraries to be4.0.0-SNAPSHOTathttps://github.com/KhronosGroup/KTX-Software/blob/3038f7cae9c875c0060ebb3d809e031ce088707f/interface/java_binding/pom.xml#L11. I preliminarily and manually changed this to 4.3.3-SNAPSHOT in this branch, but we'd have to check where this version will come from when all this is to be automated.
revision is overridden when Maven is run because the command run by the CMakeLists.txt is
${MAVEN_EXECUTABLE} --quiet -Drevision=${PROJECT_VERSION} -Dmaven.test.skip=true package
PROJECT_VERSION is set to KTX_VERSION in the top-level CMakeLists.txt and KTX_VERSION is the version with major.minor.patch.
So version naming is already handled.
and (if the Travis->GitHub transition progresses) the packaging of the natives into JARs. Ideally, this could include the upload of the JARs to Sonatype/MavenCentral (maybe not strictly required, but certainly desirable).
The Travis->GitHub transition is not essential. As I wrote, Java can use exactly the same techniques as we are using for python to package the native libraries into JARS and upload them to Sonatype/MavenCentral. Take a look at the workflow file I pointed you at.
@javagl I am preparing the next release. As part of that I am creating a new GitHub Action to build just the documentation. This has lead to the following questions and requests regarding the Java wrapper documentation.
- To build the documentation do you have to do the whole Java/JNI build or is there a way to build just the documentation?
- The Java documentation is not linked into the overall documentation package,
pkgdoc/packageDoxyLayout.xml. You will see the Python and JSWrapper documentation there. Please add it. - For ease of deployment, it should be copied to (or built in) a subdirectory of
/docs/html. This is so we have a documentation package for all of KTX-Software. Please make sure this is so. I haven't built Java/JNI recently locally. This may already be the case. You can still deploy just the Java documentation somewhere if you wish.
I haven't had the chance to make substantial progress here, but short notes:
- To build the documentation do you have to do the whole Java/JNI build or is there a way to build just the documentation?
It is possible to isolatedly build the documentation, using a plain maven call like mvn javadoc:javadoc.
- The Java documentation is not linked into the overall documentation package, pkgdoc/packageDoxyLayout.xml. You will see the Python and JSWrapper documentation there. Please add it.
- For ease of deployment, it should be copied to (or built in) a subdirectory of /docs/html. This is so we have a documentation package for all of KTX-Software. Please make sure this is so. I haven't built Java/JNI recently locally. This may already be the case. You can still deploy just the Java documentation somewhere if you wish.
Both should be relatively easy. By default, mvn generates the outputs in the target subdirectory of the project, but it should be possible to either 1. generate it elsewhere or 2. copy it from there to the right place (like docs/html).
I'll have a look at all this when I pick up the work on this PR again.
I'm ready to prepare the next release. I would like the Java documentation to be included in the documentation package that will built when Java/JNI is included in the build configuration. That means
- Copy the built documentation to a subdirectory (e.g. "java") in
<build>/docs/html. - Updating
pkgdoc/packageDoxyLayout.xmlto include the java docs. - Most likely updating other
pkgdoc/*DoxyLayout*files so the Java docs show up in their tables of contents.
Note that the pkgdoc/*DoxyLayout* stuff is all a bit of a hack. I have not found a good way to put a front end on a documentation collection with Doxygen. At this point I want to do the minimum so I can build the release. We can look at improving it later.
@MarkCallow I'll likely not be able to allocate enough time for sorting out the documentation before the next release is due. I did a very quick pass (the state is in https://github.com/javagl/KTX-Software/tree/include-java-docs ), but I have not used CMake/Doxygen extensively before. (I mean... I'm not even sure whether the documentation should be generated with javadoc or with Doxygen. The latter can be applied to Java as well...). And given some complexities and unknowns in these ...doxyLayout.xml files, tweaking all this so that it eventually appears properly on the public docs page is something that I need more time for.
@MarkCallow I'll likely not be able to allocate enough time for sorting out the documentation before the next release is due.
Do it when you have time. The current release does not have documentation for the Java wrapper so it is not a major problem to also not have it in the upcoming release.
Good question about whether to use Javadoc or Doxygen. Note that use of Doxygen will not remove the need to hack those DoxyLayout files.
The latest main has a fix for the Windows build errors you are now seeing. The errors are due to an update to ClangCL which adds a new warning about mismatched function types in a cast. The fixes in main disable the warning. I'd prefer to fix the code to not get the warning but at present have no idea how to. I'll be grateful for any clues.
Do it when you have time.
And not one second earlier 😁
Good question about whether to use Javadoc or Doxygen.
I don't have a strong opinion, but mainly because I haven't used Doxygen for Java yet. One could argue for javadoc because Java users are (more) familiar with that, or for Doxygen because it will likely cause a more consistent appearance on the website. (The javadoc-based results will be part of the Maven release anyhow...) I'll have a look and maybe share some previews if possible.
The latest main has a fix for the Windows build errors you are now seeing.
I'm not deeply involved here, and would first have to check: Is that cast safe? (A lot of context knowledge is required surrounding some of the OS-specific parts and some of the #ifdef's there). I think that one can sometimes hackily avoid such warnings by casting the source to some void* temp, and then casting that temp to the desired target, but ... if that's not safe, bad things can happen...
I have a question about this PR is this already on maven because I am making something in Java and in my build.gradle I need to use KTX tools but it isn't available on maven yet? I hope this is completed soon but no rush.i was just curious if a beta of this pr is on maven.
@Thunderrock424242 No, the KTX library is not yet available in Maven Central. This PR aims at preparing that on a technical level. There are still several questions to be sorted out in that area (e.g. for the build process). After that, there will be some "organizational" questions (signing and the actual uploading). I'll try to allocate some time to proceed here.
@javagl please make some progress on this. I would like to include these changes as part of the eventual 4.4.1 release. I have no date in mind but already have a collection of fixes I'd like to release.
Please note that ktxTexture2_DecodeAstc was added not long before the v4.4.0 release to expose the decoder that is part of the underlying astc-encoder software we use. If that function is not already exposed in the Java binding, I do not think it is, it should be added. Please add it to your TODO list. A separate PR for it is better than including it here.
The repositories that I currently have on GitHub are the tip of an iceberg of open-source projects that I'm maintaining, to varying degrees. All this is happening solely in my spare time, in addition to the paid work that I have to do, in order to.... you know, ... be able to buy food. (I sometimes eat food). KTX-Software (and this PR in particular) may therefore only receive only a tiny, tiny fraction of my time.
However, I do intend to wrap up this PR (when I find the time).
And I took the note about ktxTexture2_DecodeAstc, which indeed should go in as a small PR, because it's only related to the API, and not to the (Maven) depolyoment.