Pixi equivalent of conda constructor

[vigneshmanick]

Problem description

Hello Pixi devs,

Conda constructor

I use Conda constructor for creating installations that can then be used on air gapped machines (no internet).

A very simple overview of how constructor works is as follows

• Create a conda environment(multiple envs are also supported) based on the dependencies specified (environment.yml or directly in the construct.yml)
• Create installers for each OS specified (e.g linux, win, osx )
• In addition the installers also support pre post install scripts/hooks and also allow setting environment variables
• All of this data is packaged into an installer per platform which can then be run to recreated the conda installation on the target

These installations can then be used to create an offiline installation on any machine. The installation is a conda installer (customized since it has all the dependencies also packaged as part of the installation) that can then be installed using the same cli used to install miniconda.

Enhancement proposal for pixi

Suggestion/Enhancement here would be that pixi also support such a similar mechanism. From what i have looked all/most of the necessary plumbing is already there.

A rough idea of how this might work is as follows

1. Use pixi.toml to specify the dependencies and tasks
2. pixi construct creates a .pixi folder with all the dependencies per environment stored
3. an installation script similar to what pixi currently has is generated per platform with the difference being that running this will install pixi and also the envs created in the previous step
4. All this data is available as an installer/zip file
5. The installer should also run the pre/post install hooks.

There is also another tool from conda called conda pack but this doesn't offer the same flexiblity/robustness as conda constructor since conda pack more or less creates an archive out of the existing env to make it portable and doesn't offer any way of customizing. In addition conda pack is not under active development.

Thanks!

[ruben-arts]

I think this would be a great enhancement. I've heard more feature requests like this. Simply to improve the deployment question.

Please get this upvoted! :+1:

[bollwyvl]

Yep, this is a very important end state of a .conda-related workflow, and is probably a hard blocker for a number of projects adopting pixi.

Of note, this can probably already be done with the battle-hardened constructor itself as e.g. [task.installer] by pointing construct.yaml#environment at an already-installed .pixi/envs/{env-name}. I haven't verified this, but if not, it's probably a bug in pixi.

Another intermediate point would be to be able to generate the @EXPLICIT file for a given env, and use that as an input to constructor. Again, this probably would already work with e.g. ((micro)?mamb|cond)a to generate the well-known, if flawed, file.

To improve on the state of the art, an only-works-with-pixi installer would need to do one or more of the following, without regressing on any of the others:

• build substantially faster
• use substantially less intermediate space on disk
• require fewer dependencies (looking at you pillow on windows)
• install substantially faster
• not use known-broken cryptography
• generate substantially smaller installers
• be better at supporting pypi dependencies transparently (still an advantage of conda-pack)
• be bit-for-bit reproducible (given an input lockfile and installer configuration and e.g. SOURCE_DATE_EPOCH)
• provide better build metadata out of the box, e.g. an SPDX/CycloneDX SBOM
• support alternate outputs, such as .msi
• put fewer constraints on an output (constructors generally must contain a conda-like executable, and usually a python)
• provide a nicer CLI and GUI
• better support walled-garden "app stores" (buh)
• provide better native desktop/launcher icons

[pavelzw]

We developed pixi-pack which solves some of these use cases. It covers some (imo the most important ones) use cases mentioned here but not all. Feel free to check it out and see whether it covers your needs.

I personally think that this feature should belong into a different tool (like pixi-pack or if you think it should go in a different direction feel free to open issues there/create a fork) to not make pixi's CLI bloated. Also with pixi-pack's existence I don't feel for the need to reinvent the wheel here. WDYT?

With that being said, is this issue still relevant or should we close it? I could also write some documentation for https://pixi.sh that mentions how pixi-pack can be used.

[tdejager]

Documentation is always welcome @pavelzw :)

[bollwyvl]

Yeah, pixi-pack looks useful in many scenarios... but a user still has to get pixi-pack and the tarball.

Once a release lands with #1873, I'm planning to look more closely at the pixi.toml -> construct.yaml -> my-installer.(exe|sh|dmg) toolchain. This should end up being a very small amount of plumbing, so could be a good writeup.

Not having a rattler-based constructor wouldn't be as efficient as if pixi, rattler-build, and not-constructor could share the same caches and hot paths for solving, downloading, unpacking, etc.... but heck, constructor supports micromamba, and maybe could be taught to speak pixi.

One thing I haven't really looked into fully is the rust-based tauri. This is compelling as .msi is really the most adoptable output in a managed windows situation. The "native" web view piece on Linux (some gtk thing) wouldn't be my cup of science visualization tea, but would be totally sufficient for config, etc.

An interesting intersection is this example... this downloads a constructor installer at run time, but presumably it could be made fully offline-capable as well. Or include pixi-pack.

So yeah: I think continuously tested examples that end up in narrative docs will end up being the right call, as the matrix of use cases gets enormous at that end of the stack. For installers, getting to parallel, restartable execution, e.g. pixi run -j8 build-installer would be the biggest boon.

[pavelzw]

but a user still has to get pixi-pack and the tarball

xref https://github.com/Quantco/pixi-pack/issues/4

[vigneshmanick]

Yeah, pixi-pack looks useful in many scenarios... but a user still has to get pixi-pack and the tarball. +1

This issue is still relevant, my use case for constructor is a completely airgapped machine where neither conda nor other tools (pixi-pack) exist.

Just copying an environment elsewhere is, imo. not an ideal delivery mechanism since what is required is an end user usable artifact, not recreate a developers machine elsewhere.

In my case the installer that constructor produces

1. contains pre/post install scripts
2. Extra data(examples/docs) that is also unpacked next to the conda env
3. Once deployed directly provides an entrypoint that lives next to the installation that helps users to call the environment inside with all the user relevant customizations
4. An example installation structure is shown below

root:.
├───bin
    └───start # this script sets the necessary variables/scripts required for activating and calling the environement
├───python # this is where the constructor artifact is deployed
    └───envs
    └───etc..
├───OtherData
└───Examples

I still belive that pack could be part of a pixi project definition. There is pixi add , pixi build is on the way and deploy/pack will complete the entire toolchain nicely.

[pavelzw]

my use case for constructor is a completely airgapped machine where neither conda nor other tools (pixi-pack) exist

pixi-pack is a statically linked executable, you can just copy it together with environment.tar over to the air-gapped machine and it will work; no need to install anything

[wolfv]

Just fyi this was an experiment to make a self-executable mamba constructor like thing https://github.com/mamba-org/monstructor

I think something similar could work for pixi pack where it embeds itself in front of the compressed data :)

[vigneshmanick]

my use case for constructor is a completely airgapped machine where neither conda nor other tools (pixi-pack) exist

pixi-pack is a statically linked executable, you can just copy it together with environment.tar over to the air-gapped machine and it will work; no need to install anything

• is it possible to also bundle pip dependencies to this artifact ?
• nodejs can be installed via conda but the npm dependencies have to be installed manually, will these also be transferred into the pixi-pack tar or should they be installed manually after unpacking?

[pavelzw]

is it possible to also bundle pip dependencies to this artifact

not at the moment, see https://github.com/Quantco/pixi-pack/issues/38 although i would suggest to build a conda package out of them anyway and put it on conda-forge as this tends to be the more stable approach from my experience

nodejs can be installed via conda but the npm dependencies have to be installed manually, will these also be transferred into the pixi-pack tar or should they be installed manually after unpacking?

pixi-pack only installs the conda packages, so the npm dependencies will not be included and need to be installed separately in some way. i would suggest to just package them into a separate conda package and use --inject, see https://github.com/Quantco/pixi-pack/tree/main/examples/webserver

[bollwyvl]

Yep. "Just" turning conda packages into widely-respected (un)installer formats recognized by users (and their IT policies) is already a big lift, but "make anything relocatable," is... almost intractable, and indeed the purpose of the conda approach in the first place.

Even before pixi build is a thing, a slightly-more-than-minimum viable constructor example should likely show how to inject a simple, pixi run rattler-build output into an installer build with little more than cat.

[pavelzw]

I added some documentation in #2220 for pixi-pack. I personally feel like a more involved constructor is out-of-scope for pixi to not make the UX of pixi as bloated. This was the original reason why I created a separate tool instead of contributing the code to pixi. Since the (imo) most important use cases are already covered by pixi-pack, how about we close this issue as completed? WDYT @ruben-arts?