Yggdrasil icon indicating copy to clipboard operation
Yggdrasil copied to clipboard

Make `YGGDRASIL_DIR` an "official" variable set in recipes

Open fingolfin opened this issue 1 month ago • 1 comments

A bunch of recipes now do something like this:

const YGGDRASIL_DIR = "../../.."
include(joinpath(YGGDRASIL_DIR, "fancy_toys.jl"))

I think this is an excellent idea, making it a tad easier to move recipes into subdirectories, and generally making it clearer what the intent is. Increased consistency also helps people trying to get into the Yggdrasil ecosystem, which is getting more complex as time goes on. This was also motivation for making PR #12554 which changes even more to use follow this pattern.

Now IMHO it would be even nicer if the recipes did not have to set YGGDRASIL_DIR.

I wonder: can't we somehow come up with a way to set this variable before the script is executed?

Of course any solution needs to take into account that one should still be able to build recipes locally, and not just inside the BuildKite runners driving Yggdrasil. In the latter, it would of course be easy to set YGGDRASIL_DIR, but in the former a user could run julia build_tarballs.jl anywhere; and in fact, users might use BB entirely outside of Yggdrasil (e.g. a company who uses BB to make binaries of their own proprietary tech).

Here is one idea: we add a helper locate_yggdrasil(dir) to BB. Recipes call it like this:

const YGGDRASIL_DIR = BinaryBuilder.locate_yggdrasil(@__DIR__)

That function then could recursively look at the given directory and its parent directory, until it finds one it considers an "Yggdrasil root". The decision for that could be made based on the presence of a magic file, say .yggdrasil_root; or on some existing file such as fancy_toys.jl. The former has the advantage that any "alternate yggdrasil" could also benefit from it... Though then perhaps leaving "yggdrasil" out of the filename would be even better (and then locate_yggdrasil could be renamed, too -- suggestions welcome)

fingolfin avatar Nov 18 '25 11:11 fingolfin

Personally, I don't think we should be caring about build scripts located outside of Yggdrasil trying to find Yggdrasil's root. If someone is building a script that needs one of these scripts, they should either fork Yggdrasil and keep theirs inside their fork, or take the scripts into their own build system.

As for how we do this, I don't think a variable would work because while we could set it in Buildkite for our builders, it won't be set locally by the julia build_tarballs.jl incantation. So we would need to add a helper function to either BinaryBuilder or BinaryBuilderBase.

I think the detection logic should be tiered like this:

  1. Environment variable BINARYBUILDER_YGGDRASIL_DIR.
  2. Use the active project path as a starting point (e.g., look at if Base.active_project() is part of a Yggdrasil tree, which would probably mean is the directory immediately above it the yggdrasil root directory).
  3. Work upwards from the recipe seeing if we can find the Yggdrasil root.

imciner2 avatar Nov 18 '25 15:11 imciner2