Add support for pixi lock files

[matthewfeickert]

conda-lock itself hauls in... a lot of dependencies, so may not be a good candidate for the "base coat" environment. micromamba, already present, is certainly up to the task of consuming both formats... though pixi very well might end up "winning" for this use case.

Originally posted by @bollwyvl in https://github.com/jupyterhub/repo2docker/issues/1312#issuecomment-1775904081

This was already touched upon in Issue #1312 a bit (in the above), and now also with https://github.com/conda/conda-lock/issues/615 being open, but it would be quite nice if pixi lock files could be supported formally by repo2docker. The ability to do cross-platform solves in parallel already brings the accessibility barrier for projects down quite a bit, and then being able to reuse that environment solve for repo2docker is very attractive from a user perspective.

I appreciate that supporting a new standard is a lot fo work, and @bollwyvl has already shown in Issue #1312 that these sorts of changes are hard. I'm asking as I think it would be very cool!

[minrk]

I've been thinking about lock files on and off for a long time, specifically about how to reconcile a "fully specified" environment with a lockfile with the repo2docker approach of a 'base' environment with jupyter-server, etc. about which we have some assumptions for how to launch, which gets updated with a user environment. The problem generally is that repo2docker's design is meant to enable folks to write environment specifications that don't consider Jupyter, but we make it work by adding the required dependencies. This isn't friendly to using lockfiles (and is generally a problem class of its own).

There are 3 basic approaches to this that I can think of:

1. install base env, then 'update' with lock, like we do currently with 'loose' environments (could cause conflicts, doesn't really satisfy lock specification)
2. install lock first, then try to minimally add 'required' packages (doesn't satisfy lock specification, may break things in different ways)
3. leave lock unmodified, in which case required packages may be absent

So far, I think the best approach is the last one, to respect lockfiles exactly as-is, and install nothing 'automatically', with the following logic:

1. inspect lockfile, and if jupyter-server is present (or specify some other condition), use it as the default env
2. if jupyter-server is not present, install a kernel-only env, like we do for 'old' Pythons (< 3.7).

The case this will not handle is if no kernel package is in the lockfile, there will be no kernel installed in the locked env. I don't think there's any way for us to install ipykernel that won't make a mess at least sometimes. Usually pixi add ipykernel would work! But it's a lot harder to be sure and a lot harder to reason about "your lockfile is modified slightly sometimes" than "your lockfile is respected, but you might need to add packages to it for it to work with repo2docker".

This approach would be generally applicable to lockfiles (conda-lock, pixi, anything else we decide to support). There are trade-offs for everything, but I think this is the one that's easiest to deal with and most consistent with what lockfiles are meant to do.

[matthewfeickert]

I'll note that while https://github.com/conda/conda-lock/issues/615 is still not resolved, here is a recipe that I've found works with Pixi and Binder:

Create the following binder/postBuild configuration file in your repository

#!/bin/bash

set -ex

# Install Pixi
curl -fsSL https://pixi.sh/install.sh | sh
. ~/.bashrc
# Sourcing ~/.bashrc doesn't seem to pick up changes, so manually set PATH
export PATH="~/.pixi/bin:${PATH}"

# Export workspace to conda explicit spec
pixi workspace export conda-explicit-spec --platform linux-64 binder/

# Install environment
mamba install --name notebook --file binder/default_linux-64_conda_spec.txt

and upon Binder build your default environment in the Pixi workspace will be installed into the notebook environment that Binder uses.

A potential problem here is that if you have Jupyter related things in your workspace you'll be asking to overwrite what Binder supports already in the existing notebook environment, which could potentially cause problems.

[VeckoTheGecko]

A potential problem here is that if you have Jupyter related things in your workspace you'll be asking to overwrite what Binder supports already in the existing notebook environment, which could potentially cause problems.

Another thing as well is that this doesn't support things that are installed from source using Pixi. I'm not sure if Binder generally supports things installed from source? (e.g., via pip install .)

(base) 🦎parcels   pixi-binder[$?] ❯ pixi workspace export conda-explicit-spec --platform linux-64 binder/
Error:   × Conda source packages are not supported in a conda explicit spec. Specify `--ignore-source-errors` to ignore this error and create a spec file containing only
  │ the binary conda dependencies from the lockfile.

Maybe thats beyond scope for this issue...

[matthewfeickert]

Another thing as well is that this doesn't support things that are installed from source using Pixi. I'm not sure if Binder generally supports things installed from source?

I would normally put that in a binder/postBuild script anyway. So your source install would just be added as a line there. Having a source install where you aren't providing a hash doesn't belong in a lock file anyway, so that doesn't seem to be an issue here, right?

I think that your point is just that a conda explicit spec file doesn't support non-conda packages. That's indeed a limitation, which I don't think is going to be overcome in the conda spec world, and would need to be achieved through actually providing Pixi support.