feat(build): load variants from packages

[wolfv]

Problem description

Today, we have the ability to specify variants for pixi-build on the workspace level.

[workspace.build-variants]
python = ["3.11.*", "3.12.*"]

However, commonly we would want to load variants from a shared variants.yaml file, or even from a package (like the conda-forge variants) in order to guarantee compatibility ecosystem wide.

For this reason, I propose to add a configuration that would look something like this:

[workspace]
build-variants-files = [
    "./corp-pinning/config.yaml"
]

[workspace.build-variants-packages]
conda-forge-pinning = "2024.01.01"

This would enable to:

1. load the conda_build_config.yaml from a file. The package resolution follows regular conda package resolving, but instead of creating an environment / installing the file we just place it in the package cache and find the ./conda_build_config.yaml or variants.yaml file at the root of the package. This should ideally also work for source packages.
2. Pointing directly to a file: relative to your workspace pixi.toml you can point to a file that we load from YAML. If the file is called conda_build_config.yaml we use the conda_build_config parser, otherwise the new "variants.yaml" rattler-build parser.

The order of the files matter. It follows the same prioritization order as e.g. channels (first file overrides configuration from the ones that appear later in the list).

[baszalmstra]

This looks great! Things to consider:

• We also have to lock this into the lock file to make sure it reproduces.
• When it's locked, how do we upgrade it?
• Do we properly parse the conda-build configuration file already? I suppose it includes selectors and similar elements?

[wolfv]

Maybe we can make the files part of the input hash for the metadata? That would then work whenever they change.

I think we could require packages to not have any dependencies so that we can just pull them in the package cache and read them from there. Or maybe setting up an actual env is simpler? Idk - would love to hear what you think @baszalmstra

[wolfv]

Implementation wise, should we implement this as part of the command dispatcher? Maybe we should start with an implementation that loads from a file, and later only implement the version that loads from a package as there are a few more implications (e.g. when depending on a variant package from source).