Allow "meta package" builds for pixi build

[jackm97]

Problem description

Consider a workspace with multiple packages, but no specific "main" package, in contrast with the example in the documentation. One can imagine a situation where multiple teams work on different packages that are all equally important and only when are they all together do they constitute a complete product. This is the case for my current workplace.

The top-level workspace manifest might look like

[workspace]
name = "Product Packages"
platforms = ["win-64", "linux-64"]
preview = ["pixi-build"]
channels = [
  "https://prefix.dev/pixi-build-backends",
  "https://prefix.dev/conda-forge",
]

[system-requirements]
libc = { family = "glibc", version = "2.34" }

[dependencies]
package1 = { path = "./packages/package1/pkg-spec" }
package2 = { path = "./packages/package2/pkg-spec" }
package3 = { path = "./packages/package3/pkg-spec" }

Now, this is great if all I want to do is install the packages and tar the env using pixi-pack, which will likely be a common use case for sharing reproducing environments for bugs and such.

But we have a lot of work that can't be run locally and needs to be submitted as a job to our CI infra. We have an artifactory that or CI manifests already support, which would make using conda packages a breeze.

So it would be nice if it was possible to do pixi build so all the .conda packages are dropped in the working directory allowing for upload to the artifactory.

I imagine this would be easiest with some sort of meta package type, where it just aggregates individual package builds. Maybe even a new backend? I'm not sure...

[jackm97]

To add, I'm thinking the manifest would look something like this:

[workspace]
name = "Product Packages"
platforms = ["win-64", "linux-64"]
preview = ["pixi-build"]
channels = [
  "https://prefix.dev/pixi-build-backends",
  "https://prefix.dev/conda-forge",
]

[system-requirements]
libc = { family = "glibc", version = "2.34" }

[dependencies]
product-package = { path = "." }

[package]
name = "product-package"
version = "1.0.0"

[package.build]
backend = { name = "pixi-build-metapackage", version = "0.1.*" }

[package.run-dependencies]
package1 = { path = "./packages/package1/pkg-spec" }
package2 = { path = "./packages/package2/pkg-spec" }
package3 = { path = "./packages/package3/pkg-spec" }

[ruben-arts]

We are planning to create a pixi publish command which would build all the [packages] in a workspace and upload them "somewhere" I think that command would fit this requirement. It should then also create a lockfile that uses those packages instead of the local once, so you can let anyone install it as built conda packages.

If I understand your request correctly, this doesn't need a new backend but just some functions to comfortably do this. The current workaround would be:

1. run pixi build in all packages. This will create a .conda file in all those directories.
2. upload them to a channel using pixi upload
3. change the dependencies in the workspace manifest to depend on those specific packages instead of the local version.
4. pixi lock the lockfile

You now have a pure conda package installation and don't need to build the packages on every machine.

[jackm97]

@ruben-arts, that definitely covers some of the use cases I'm thinking and overall would be really helpful.

However, it would be nice to have a way to create a single package that can install all the deps, rather than just build and publish them. This single package would have all the individual ones as deps.

But I'm realizing that use case is easily handled by rattler backend.

Also, will there be a way to depend on other source packages in the package dependencies? It's usually ideal to be able to download conda packages, but in the case that someone is working on multiple interdependent packages at once, this would be useful. This would be a common thing for us.

If not, I can create a new issue for that.

Edit: To clarify, the source dependency is required at build time, not runtime.

[tdejager]

Hi @jackm97, If you mean: https://github.com/prefix-dev/pixi/issues/2814 this is definitely on the roadmap, we hit some snags where the design will be more difficult than expected but will be revisiting that soon :)