build: new system to build dependencies locally if needed

[lgritz]

trafficstars

High level TLDR:

• If checked_find_package doesn't find a dependency (or it is not an acceptable version), it looks for src/cmake/build_<pkg>.cmake and if that exists, includes it. It can do anything, but is expected to somehow provide the dependency so that a second find_package will find it and then proceed as if it were a system install.

• I've implemented these scripts so far for Imath, OpenEXR, OpenColorIO, fmt, and robin-map, that download, build, install the package in OIIO's build area. More to come later?

• This is really simple with a new build_dependency_with_cmake macro, much simpler than ExternalProject_Add, as I've seen it used elsewhere.

• Just look at any of the new build_blah.cmake files to see how simple it is for each new dependency we set up this way.

• By default, pre-installed packages it can find always take precedent over building locally. So if you have all the dependencies already installed, none of this should behave any differntly than before. But there are variables that let you override on a package by package basis, giving the option of never building locally, building locally if the package is missing, or forcing a local build to always happen.

---

Various details:

A bunch of cmake things (including checked_find_package) have been moved into a new file, dependency_utils.cmake.

build_Imath.cmake, build_OpenColorIO.cmake, build_OpenEXR.cmake, build_Robinmap.cmake, and build_fmt.cmake implement local builds of those packages. They're very simple, and lean heavily on common infrastructure of build_dependency_with_cmake, which also can be found in dependency_utils.cmake.

Robinmap and fmt are extra simple because we use them as header-only libraries.

For Imath and OpenEXR, I build them as static libraries, so they will be incorporated into libOpenImageIO (and/or _Util) libraries to be totally internal to them, there should be no symbols exposed outside our libraries. This should mean that the resulting libOpenImageIO should be perfectly safe to link in an application that also links against OpenEXR, Imath, or OpenColorIO, even different versions thereof, without any interference. Note that none of those packages are used in our public APIs, only internally.

OpenColorIO was a little trickier. It builds its own dependencies as static libraries that are internalized, but OCIO itself is a dynamic library. So we end up having to make it part of our install, but I use OCIO's build system to make a custom symbol namespace and a custom library name, so it still should not interfere with any other OCIO linked into the application.

We'll see how it goes for furture dependencies we want to add. The header only, static libraries incorporated and hidden, and dynamic library but renamed and with custom namespace, are all techniques that work well. I'm not sure I'd advocate doing local builds of any dependency that we can't incorporate in one of these ways, but I guess we'll cross that bridge when we get to it.

New option_utils.cmake has two handy new utilities: set_cache() is much like the built-in set() when making a cache variable, and set_option() is much like build-in option(). The biggest difference is that both allow an environment variable of the same name, if it exists, to supply the default value. This is something that cmake does with many of its own controls, like CMAKE_BUILD_TYPE, but does not make any provision for built-in set() or option() let users do it.

checked_find_package() has moved to dependency_utils.cmake, and has been enhanced to take several new options, and also so that if the enclosed find_package() fails and there is a src/cmake/build_PKG.cmake, it will run it to build the dependency itself in the build area. If that build_PKG sets a variable called PKG_REFIND, it will try find_package again to find the one just built.

build_dependency_with_cmake() is given a git repo and tag, and basically clones the repo, checks out the tag, configures, builds, and installs it (all in our own build area).

[lgritz]

I've marked this as a draft because I noticed that I'm having trouble building opencolorio in this manner on Windows. Needs some minor fixes, but then I'll remove draft status. But I'm confident this is 95% in final form, so by all means start taking a look if you're curious about the approach I'm trying.

[ssh4net]

Is it mandatory from now on to build everything as static libs, or OIIO, and required libs as before can be built as dynamic libs?

[fpsunflower]

I like the idea of making the build easier -- at the same time, it feels to me like this is kind of reinventing the wheel. I worry that this setup will be brittle and need constant tweaking (though maybe that is par the course no matter what we do).

For someone new to the project, I think it would be nicer to see calls to the ordinary cmake find_package than discover we've rolled our own flavor that they need to study. This would likely be easier for package maintainers as well.

As an alternative -- would it be worth seeing how many custom "find*.cmake" we can actually remove if we rely on new enough versions of the target libraries (and cmake itself)? How close to a fully "modern cmake" setup could we get?

[zachlewis]

I've only tested on linux so far, but it works like a dream. I can't tell you how much easier this makes things for us...!

[fpsunflower]

I did a bit of reading and some people suggest using the FetchContent family of functions in cmake.

Did you investigate those? Is there a reason why you rolled your own version?

I'm curious what the pros and cons are. From a cursory glance, it seems to be designed for this exact problem.

[fpsunflower]

Another cmake feature that came up when reading up a bit on this was ExternalProject. I'm not sure I fully grok the differences between that and FetchContent but it seems like another option that might be relevant here.

Likewise there is also ExternalData which could be handy to use for some of the testsuite data (which is a bit clunky to download and setup manually).

[lgritz]

@ssh4net:

Is it mandatory from now on to build everything as static libs, or OIIO, and required libs as before can be built as dynamic libs?

None of this is mandatory. It only kicks in if important dependencies are not found, or are not adequate versions. I prefer static libs for this particular use because I don't want OIIO to need to also install dynamic libs of the dependencies, which might interfere with system libraries.

@zachlewis:

I've only tested on linux so far, but it works like a dream. I can't tell you how much easier this makes things for us...!

Thanks! I'm still working out some issues for the Windows CI, but that's less that it's difficult and more than I haven't had the time to chase it down in the last several days.

@fpsunflower:

This is my 4th or 5th rewrite of this, including doing full implementations of both the ExternalProject and FetchContent families of functions (and the CPM project as well). Every one had its own set of awkward tradeoffs, and I didn't find any of them as simple and clean as what I've done here. You can see those other implemented variously in OpenEXR (which uses FetchContent, I think, to get Imath) and OpenColorIO (which uses ExternalProject for all its dependencies). Both seem to require considerable setup before the ExternalProject/FetchContent, and considerable extra work after, to fix up everything just right. Whereas I feel like my stab at it is a really minimal way to get EXACTLY what you would get if it found the project externally -- in fact, after doing a build, it just does another find_project so it really does read the exported cmake files of a clean build of the dependency (none of the others work that way).

Here is OpenColorIO's code to build OpenEXR when not found: https://github.com/AcademySoftwareFoundation/OpenColorIO/blob/main/share/cmake/modules/install/InstallOpenEXR.cmake It's based on ExternalProject_Add and you can see how much ridiculous cruft must surround it. Every one of their dependencies needs a similarly impenetrable file. Compare that to my build_OpenEXR.cmake in this PR, and you can perhaps understand how I arrived at this solution after trying the others.

Note that the general setup I have where it looks externally, then if not found, runs whatever is in the build_blah.cmake, can do anything in that build file. Nothing stops one of the build_blah files from doing ExternalProject or FetchContent or CPM or anything else if it wants. So it's trivial to switch the methodology of that part if we decide down the road that I'm just totally misunderstanding those and they are easy after all. I allow for the possibility that I (and maybe OpenColorIO's authors and those of every other project I looked at) have misunderstood ExternalProject/FetchContent and made it overly complicated. I could be wrong.

[lgritz]

One issue with FetchContent/ExternalProject is that it's very awkward if the dependency is cmake based, and moreover must have its cmake build system carefully written to be safely included as a subproject. My scheme makes no such assumptions.

[fpsunflower]

I see. That's disappointing to hear. From the advertising it sounded like those commands were solving the exact same problem in a built-in way that would (hopefully) be more familiar to anyone doing the build and more likely to work across platforms out of the box.

From what I was reading -- the whole point of ExternalProject is that you get to invoke cmake again and control its environment exactly. I'm not sure why it would require anything special from the target project (in fact it claims to work with any build system, not just cmake). Once the recursive cmake configure,build,install succeeds, it should be in the build folder as if it had been installed on the system and should be able to be picked up as such. The only catch is that you are getting a completely independent cmake session, so any arguments you want to forward (like compiler, build type, etc...) you have to pass in manually. That could definitely get verbose, although I guess some of that could be automated.

FetchContent on the other hand, definitely seems a bit more fragile because if I understood it correctly, all it does is it adds the fetched projects' targets to yours. So I could definitely see that a not-so well behaved project could make a mess. But for well-behaved project with modern cmake throughout, you get (in theory) a clean dependency graph of all the targets and they all execute within the same cmake session (for better or worse).

In any case, didn't mean to derail the conversation -- it could be that over time the official cmake way of doing things will mature and will be able to be swapped in as you said.

[lgritz]

So here is my understanding:

FetchContent just gets stuff and puts it in your tree. It doesn't invoke any build as far as I know, you'd need to add all that logic. But it does it immediately at configure time, so may be ok for a header-only library. I believe that the integration between FetchContent and find_package is fairly new, and we can't count on a cmake that recent unless we make our cmake minimum much higher than any of the other related projects.

ExternalProject gives you the works, including build steps, etc., but the problem is that the download doesn't happen until the build step itself, which is problematic for two reasons -- first, many distros and build setups have firm rules against needing internet access beyond the configure step, and second, it means that you can't use the targets or anything else you would've learned from the config files for the rest of your build step.

I also tried CPM which at first appeared to be exactly the wrapping and simplification I wanted, but I found that is really only works for very well structured cmake projects and I had trouble getting it all quite right with the interdependencies of OpenEXR, Imath, and OpenColorIO.

What I found trying to explore these in detail is that they are pretty good under certain conditions: (1) well structured dependencies that have already made themselves subproject-friendly, (2) independent dependencies, so you don't need to worry much about a package that is both a direct dependency of you and also of other dependencies, (3) your aim is unconditional building of dependencies rather than a somewhat complex set of rules we have about preferring system installs. I just didn't find the level of control I preferred, with all the little features I wanted, in a way that it could be expressed compactly for each dependency and didn't require me to have to fight to find exactly the right secret incantation to make it work. So I wrote my one from scratch that covers exactly what we need, with a very simple use-site notation.

I dunno, I could be wrong. But I couldn't make them work without feeling like I was jumping through some ugly hoops. What I would like to do is get this scheme integrated, and then it is easy for us to experimentally swap out any individual dependency's build_dependency_with_cmake call with a FetchContent or ExternalProject to see if that works better.

[lgritz]

Updated and changed from draft status to "ready to review".

I finally have it working on all platforms including Windows (though with the compromise of using dynamic libs on Windows for now -- maybe somebody who knows more than me can return later to try making it work with static libs on Windows also, but this is fine for now).

So at this point, all of Imath, OpenEXR, OpenColorIO, fmt, and robin-map will download and build (at fairly modern versions) if not found on the system at OIIO-build time or not as recent as our minimum required versions. When not on Windows, they all do so with static libs, so they don't expose the rest of the system to any libraries or symbols that may conflict with other versions of those dependencies. On Windows, it has to use dlls for now, but do give them custom library names and symbol namespaces so that, I hope, they don't conflict there, either.

Future expansion might lean toward increasing set set of package for which we add the capability to automatically build if not found.

[cedrik-fuoco]

Hello! Just wondering what are the reasons that Conan or Vcpkg got dismissed as a solution to manage dependencies?

[lgritz]

Externally built dependencies, or automatic locally built dependencies?

For external, we don't dismiss either. In fact, we us vcpkg on our Windows CI, and a little bit of Conan on our Linux CI (for some packages). The "local build" facilities are for when they aren't found externally using whatever dependency management the user or their system has set up.

I have to admit that I'm not crazy about vcpkg in practice for the CI because, as far as I know, there are no cached binaries, no easy way to have it build only the mode you care about (e.g. for our CI, I want to build only release mode, but I seem to have no choice but to pay to burn GHA minutes for vcpkg building debug even though I'm not going to use it), and there's no fine control over which version of each dependency I get or what other build-time controls are used. I'm not sure how (or if) those can be overridden when invoking vcpkg, without doctoring the individual recipe files of ever dependency.

But if you're happy with whatever version of the dependencies vcpkg is pegged to, and the build options it uses, and don't mind that it'll build both debug and optimized (which is fine if you're going to do that build once and then use it for weeks or months), it's totally fine.

Conan -- Again, nothing wrong with it. I'm not experienced enough with it to know if it's any better in terms of easy command-based overriding of version or build options of the dependencies without doctoring recipe files or whatnot. I did look at it potentially as the basis of this PR, and to be honest, I don't remember the specific reason I chose to mostly roll my own. I think maybe it just didn't give me all the controls I needed, or maybe it was my fault and I didn't investigate deeply enough, or perhaps I thought I could cobble something together faster than it would take me to deeply learn the Conan way.

I wish I had a more principled answer to explain. But the long and short of it was that I looked at several solutions: cmake ExternalProject_Add, CPM, Conan, and some others. I found strengths and weaknesses to them all, hard to decide among them, and in the end I broke the paralysis by taking a shot at writing the dead simplest solution I could think of to exactly solve our use case with no additional complexity or cognitive load.

[cedrik-fuoco]

Thanks for your answer! I think the complexity and cognitive are very valid reasons. My intention is not to change your mind, but I was curious on the thoughts behind the choices. I think that needing finer control with vcpkg or Conan is a common issue.

I have to admit that I'm not crazy about vcpkg in practice for the CI because, as far as I know, there are no cached binaries, no easy way to have it build only the mode you care about (e.g. for our CI, I want to build only release mode, but I seem to have no choice but to pay to burn GHA minutes for vcpkg building debug even though I'm not going to use it), and there's no fine control over which version of each dependency I get or what other build-time controls are used. I'm not sure how (or if) those can be overridden when invoking vcpkg, without doctoring the individual recipe files of ever dependency.

You can use the default triplets (or the community triplets) to build in Release or Debug only. That would definitely save you those GHA minutes.

I don't recall if it is enabled by default, but vcpkg do have a local binary cache that can save you a lot of build time.

But if you're happy with whatever version of the dependencies vcpkg is pegged to, and the build options it uses, and don't mind that it'll build both debug and optimized (which is fine if you're going to do that build once and then use it for weeks or months), it's totally fine.

I had similar issues regarding transient dependencies. For example, if you have multiple dependencies that uses zlib, I don't recall a way to control the version of zlib for all your dependencies - But you can control the version of your direct dependencies using manifest mode (within the versions that are made available by the package).

Conan -- Again, nothing wrong with it. I'm not experienced enough with it to know if it's any better in terms of easy command-based overriding of version or build options of the dependencies without doctoring recipe files or whatnot. I did look at it potentially as the basis of this PR, and to be honest, I don't remember the specific reason I chose to mostly roll my own. I think maybe it just didn't give me all the controls I needed, or maybe it was my fault and I didn't investigate deeply enough, or perhaps I thought I could cobble something together faster than it would take me to deeply learn the Conan way.

I've worked with Conan for awhile and while I like the concept of it, it can feel like you don't have all the low-level control that you would like sometimes (e.g. the way the Cmake generators are implemented)

With Conan, you can control the version of your direct and transient dependencies - which is a big plus over Vcpkg, but it does have a bigger cognitive load.

[lgritz]

You can use the default triplets (or the community triplets) to build in Release or Debug only. That would definitely save you those GHA minutes.

@cedrik-fuoco Could you elaborate? What do I need to do in particular to a line like

vcpkg install tiff:x64-windows

to make it only build the release version?

[cedrik-fuoco]

You can use the default triplets (or the community triplets) to build in Release or Debug only. That would definitely save you those GHA minutes.

@cedrik-fuoco Could you elaborate? What do I need to do in particular to a line like

vcpkg install tiff:x64-windows

to make it only build the release version?

vcpkg install tiff:x64-windows-release or vcpkg install tiff --triplet x64-windows-release (same behavior, different syntax) @lgritz

It uses the Community triplets, but they come with Vcpkg: https://github.com/microsoft/vcpkg/tree/master/triplets/community

[lgritz]

@cedrik-fuoco Thanks, that's very helpful!

[lgritz]

So, here is where we are, I think. This PR basically consists of two parts, both conceptually simple:

1. When we search for a dependency PKG and it's not found, now it knows to check for src/build-scripts/build_PKG.cmake, and if that exists, it will include that file and run the code therein.

   This is fully general, the file can do literally anything, though the baseline assumption is that it either (a) does something to directly add the targets for the dependency to the local build of OIIO, or (b) does something that will allow a second attempt at find_package to succeed and we can then go on our way exactly as if it had been found "externally" the first time.

   There are many possible approaches one could take in those files: ExternalProject_Add, FetchContent, CPM, Conan, many flavors of roll your own. It literally doesn't care, as long as whatever it does permits either option (a) or (b) above.

2. I've taken a stab at making such "build" files for OpenEXR, Imath, and OpenColorIO, which in my current implementation are the simplest "roll your own" variety I could get away with, utilizing some utility functions I wrote that make it super easy to download a GitHub-based project that uses CMake, run the configure and build steps, and end up with a private install in OIIO's own build directory, which is then used to continue our build.

That's really it. Note that item 2 is not set in stone. At any point, we could improve the methodology, switch to one of the other methods, or mix and match with each additional dependency choosing the best/simplest approach to do a local build of itself.

Also note that by default, "external" (on the system already) installations of the dependencies will always be used if found (and in the acceptable range of versions), so none of this local build stuff kicks in at all except for dependencies that appear to be missing at build time.

I don't think it's possible for this to break or change anybody's setup who already has the right dependencies installed in their build environment.

I'd love to get approvals for this PR and do an initial merge, and then we can continue to refine and add more dependencies over time. My goal is that by the time we release OIIO 3.0, we'll have worked out how to make these builders for most or all of the major dependencies, so that more of them can be effectively treated as required/assumed dependencies, while decreasing the burden on downstream users/builders to have to worry about pre-building dependencies.

[lgritz]

We discussed this at yesterday's TSC meeting, and the consensus seemed to be that maybe the right default should be NOT to automatically build the missing dependencies, for fear that some downstream users may require specific dependency versions (or maybe customized builds of those dependencies), think they have those pre-installed, but if our build system doesn't find them and the automatic builds kick in, they may inadvertently get versions different than their requirement. Lee advocated for the initial OIIO build to fail if dependencies were missing, but to give a clear message saying how to turn on the option to do the automatic dependency building.

So I've amended the PR again: Make the default be to NOT build any local packages unless explicitly instructed. (Except for robinmap and fmt, which we had always automatically downloaded if missing; we continue to do that.)

But we also add a nice report after config that details which missing optinal dependencies we could have built and how to arrange it.

Here's an example of what gets printed at the very end of our build, let me know what you think @lkerley:

[lgritz]

Will merge this by end of day tomorrow if nobody has further comments or objections.

[zachlewis]

I think the OCIO-1.1 minimum thing may have broken the build system's ability to "refind" the just-automatically-built OpenColorIO-2.3.2 (i.e., with -DOpenImageIO_BUILD_MISSING_DEPS=all):

...
[18/19] Install the project...
-- Install configuration: "Release"
-- Refinding OpenColorIO
CMake Warning at src/cmake/modules/FindOpenColorIO.cmake:35 (find_package):
  Could not find a configuration file for package "OpenColorIO" that is
  compatible with requested version range "1.1...<2.9".

  The following configuration files were considered but not accepted:

    /var/folders/w7/h_1h043d5jlbq4gz7rzxdwdm0000gn/T/tmp8q4w10fu/build/deps/dist/lib/cmake/OpenColorIO/OpenColorIOConfig.cmake, version: **2.3.2**

Call Stack (most recent call first):
  src/cmake/dependency_utils.cmake:335 (find_package)
  src/cmake/externalpackages.cmake:126 (checked_find_package)
  CMakeLists.txt:176 (include)

CMake Warning (dev) at /Users/zach/.rye/tools/cmake/lib/python3.12/site-packages/cmake/data/share/cmake-3.29/Modules/FindPackageHandleStandardArgs.cmake:447 (message):
  `find_package()` specify a version range but the module OpenColorIO does
  not support this capability.  Only the lower endpoint of the range will be
  used.
Call Stack (most recent call first):
  src/cmake/modules/FindOpenColorIO.cmake:87 (find_package_handle_standard_args)
  src/cmake/dependency_utils.cmake:335 (find_package)
  src/cmake/externalpackages.cmake:126 (checked_find_package)
  CMakeLists.txt:176 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- OpenColorIO library not found 
--     OpenColorIO_ROOT was: /var/folders/w7/h_1h043d5jlbq4gz7rzxdwdm0000gn/T/tmp8q4w10fu/build/deps/dist
--     Maybe this will help:  src/build-scripts/build_OpenColorIO.bash
...

No problems building or refinding Imath-3.1.10 and OpenEXR-3.2.4...!

- All build dependencies: BZip2 1.0.8;DCMTK NONE;FFmpeg NONE;fmt 10.2.1;Freetype 2.13.2;GIF NONE;Imath 3.1.10;JXL NONE;Libheif NONE;libjpeg-turbo 3.0.0;LibRaw NONE;OpenColorIO NONE;OpenCV NONE;OpenEXR 3.2.4;OpenGL;OpenJPEG 2.5;PNG 1.6.40;Ptex NONE;Ptex NONE;pybind11 2.12.0;Python 3.9.16;Qt5 NONE;Qt6 NONE;Robinmap;TBB NONE;TIFF 4.5.1;WebP NONE;ZLIB 1.2.12

however, the report seems to think that all of the dependencies are missing:

-- =========================================================================
-- 
-- The following dependencies were not found:
--     Imath
--     OpenEXR
--     JXL
--     OpenColorIO
--     OpenCV
--     TBB
--     DCMTK
--     FFmpeg
--     GIF
--     Libheif
--     LibRaw
--     Ptex
--     Ptex
--     WebP
--     Nuke
--     Qt6
--     Qt5
--     Robinmap
--     fmt
-- 
-- For some of these, we can build them locally:
--     Imath
--     OpenEXR
--     OpenColorIO
--     Robinmap
--     fmt
-- To build them automatically if not found, build again with option:
--     -DOpenImageIO_BUILD_MISSING_DEPS=all
-- 
-- =========================================================================

(the report is lovely, btw, even if is filled with lies and deceit)

[lgritz]

Ok, I think maybe I know what's wrong. Will update.

[lgritz]

I was unable to reproduce any problematic behavior, and the thing I suspected turned out not to be broken. I pushed an update that improves the output somewhat (based on ideas I had for making it clearer while trying unsuccessfully to reproduce), but that doesn't address the problem that you describe, @zachlewis.

I don't see why putting the OpenColorIO minimum back to 1.1 should matter. (Aside: Do you by any chance have an OCIO < 2.1 installed somewhere that might be getting found and interfering with things?)

Can you try checking out a new version of this patch and trying a clean build with it? And if something goes wrong, can you please send me the FULL verbose cmake output of the whole configure and build process, maybe I can spot some clue.

[zachlewis]

Thanks, Larry --

I can pretty reliably reproduce on my end -- if I adjust src/cmake/externalpackages.cmake so VERSION_MIN and VERSION_MAX for OpenColorIO are 2.1 and 3.0 respectively, everything works as expected -- the build process either uses whatever OpenColorIO it can find, or builds its own locally.

If I don't do anything, and leave externalpackages.cmake alone, whether or not the build process first finds an existing OpenColorIO-2, the build process build its own OpenColorIO; but ultimately decides not to use it, and continues building OIIO without OCIO support.

I've created a gist here that includes the logs of three different builds, as well as the rez package.py I'm using to test building OIIO under different conditions / constellations of requirements.

• zachlewis_oiio_build_24060620.log: The build process finds the provided OpenColorIO-2.4, doesn't like it, builds its own OCIO locally, doesn't like it either, and builds without OCIO support.
• zachlewis_oiio_build_24060621.log: If I patch src/cmake/externalpackages.cmake and provide an existing OpenColorIO-2 package the build process links the found libraries
• zachlewis_oiio_build_24060622.log: If I patch src/cmake/externalpackages.cmake and do not provide an existing OpenColorIO package, the build process builds it own OCIO locally, and continues building with OCIO support.

[lgritz]

Thanks! Is that OCIO package you have installed a top-of-tree OCIO rather than any actual release? Maybe that's the key.

[zachlewis]

Thanks! Is that OCIO package you have installed a top-of-tree OCIO rather than any actual release? Maybe that's the key.

Hmm, good thought. I'd been testing with top-of-tree packages; but I just tried with an OCIO package built from the released OCIO v2.3.2.tar.gz, and the behavior is consistent with what I've been experiencing with the top-of-tree OCIO packages.

[lgritz]

I don't quite have a full reproduction as you described it originally, but I am (with OCIO TOT) able to have it not find an external OCIO, build it locally, and then not find the local one it just built! So I think I might be on the trail.

[lgritz]

I found some real problem with the way I was doing version checking. Revising and testing now. Stay tuned...

[lgritz]

OK, @zachlewis, give a try? I think I might have it.

The Windows 2022 CI failure appears to be unrelated to this PR -- it's having the same errors on the nightly test against master, and just started yesterday, so I think it's one of those occasional breakages of the github runner setups that they usually fix within a few days.

[zachlewis]

OK, @zachlewis, give a try? I think I might have it.

You did it!!! Well done, man. I appreciate the effort you've been putting in to this.