KhronosGroup
Zip format releases
Hello! Thank y'all for maintaining these tools, they've been a blessing for building our game's asset pipeline.
One problem I've run into is that we want to automatically install and manage the version of the KTX tools on developer machines. There are some great tools to manage this within repositories, like mise-en-place, but these tools expect tar.gz or zip files, generally.
Releases for Windows and macOS are only available as installers (exe and pkg) which generally aren't supported by tools like mise and are harder to automate in CI pipelines. However, the Linux and iOS builds are already available as tar.bz2 and zip formats.
Would it be possible to add a ZIP packaged version of the KTX tools for Windows and macOS? If so, would y'all accept a contribution adding that to the release matrix?
Thank you!
I am happy to hear the tools have been "a blessing."
I don't know anything about tools such as mise-en-place or the attractions for using them nor why they want tar.gz or zip. Are you aware that .pkg and Microsoft installer packages (whether .exe or .msi) can be installed non-interactively?
If the intent of using tar.gz or zip is because they want to be able to install the tools in non-standard places then you need to be aware that installing the tools and library in a non-standard place on macOS will not work. It may be possible but will require changes to the way the tools are built. If this is the intent I can provide more details.
I do not have bandwidth for supporting more configurations or package types. If this can be done with cpack in a similar way to how the Linux packages are created then I may be willing to accept a PR. In the Linux case it is simply a case of adding the other package types to the CPACK_GENERATOR variable. All the component configuration etc. is generic to all packages. Therefore there is virtually no maintenance specific to the tar.bz2 and rpm packages..
I also do not have the bandwidth to test such packages once built. Is there any way to automatically test with GitHub Actions that they can be successfully installed?
Thank you for taking the time to respond.
I don't know anything about tools such as mise-en-place or the attractions for using them nor why they want tar.gz or zip.
mise-en-place lets us define the tools needed for a project using a file within that project. It manages multiple versions of the same tool and switch between them as needed. We use it for managing versions of Node.js and tools like ripgrep without needing a single global installation.
mise can install tools from GitHub releases that have archives with a binary located somewhere within. Some tools that happen to work with mise without doing anything extra include ripgrep, sd, and nushell.
For using tools that have breaking changes, it's very handy to install them with a tool like mise to ensure that we can continue to build and work on projects even if they require older versions of projects like the Khronos Texture Tools. For example, one of our games still uses an old version of toktx in its asset pipeline, but we'd like to continue to update KTX for our main game.
Right now, for tools that we can't automatically install, we check their binaries into source control for each project. That ensures we have the same version across all developer machines and CI. Our team mostly works on Windows, but we've started using macOS as well and there it's harder to relocate binaries and have them still work.
I opened a discussion thread with the mise developers about installing the Khronos Texture Tools with mise but neither Windows and macOS nor ended up being possible. As part of that conversation, a mise contributor tried adding support for installing KTX to Aqua, which is similar to mise. Unfortunately, that was only possible for Linux with the artifacts available.
Are you aware that .pkg and Microsoft installer packages (whether .exe or .msi) can be installed non-interactively?
Non-interactive installation helps a lot for CI! I wasn't aware of it being well-supported.
If the intent of using tar.gz or zip is because they want to be able to install the tools in non-standard places then you need to be aware that installing the tools and library in a non-standard place on macOS will not work. It may be possible but will require changes to the way the tools are built. If this is the intent I can provide more details.
I appreciate this note; I hadn't thought of this. Is this because of the executable rpath? I'm not very familiar with packaging on macOS.
In our tests checking macOS binaries into the repository, they appeared to work but it's probably because the machine I tested on also has an installation of the Khronos Texture Tools at the default path.
The tools I mentioned above package for macOS, but I don't think that any of them ship any dylib files so they likely also don't configure an rpath or care about where they're installed.
If this can be done with cpack in a similar way to how the Linux packages are created then I may be willing to accept a PR.
I explored the existing cpack configuration and it seems like it should be possible. I had some trouble refitting the Windows and macOS packaging scripts to include the same kind of logic that the Linux packaging script uses. I am not that familiar with cpack.
I also do not have the bandwidth to test such packages once built. Is there any way to automatically test with GitHub Actions that they can be successfully installed?
Is there an existing strategy for automatically testing the other packages?
Testing these new packages should just involve unzipping them to a temporary directory and trying to execute each of the binaries. I would be willing to contribute that kind of automated test as part of a PR introducing the new package formats.
Thank you for explaining mise-en-place. I found yours a lot more helpful than the About page on the _mise-en-place` web site.
Non-interactive installation helps a lot for CI! I wasn't aware of it being well-supported.
You can find how-to information on the internet. We use NSIS (NullSoft Install System) on Windows (because that is all cpack supports) and .pkg on Apple. Is the issue with Windows not knowing how to do a non-interactive installation or is there more to it?
Regarding macOS I have just landed PR #1046 to make @executable_path and @executable_path/../lib the places the binaries tell dyld to search for libktx, the same as the Linux binaries. The second was previously /usr/local/lib. With this tools like mise should be able to place the binaries under a different root. They just need to ensure that the binaries and the library are in sibling directories.
I look forward to seeing your PR.
Do you still plan to submit a PR?
I am about to create a release which will include the rpath changes I described earlier. Try using the command line options for NSIS and /usr/sbin/installer (macOS) to do a non-interactive install to an alternate location when this release is available.