Facilitate docker 'layering'

[olivier-lacroix]

Problem description

To speed up docker image building, it is fairly typical to separate "things that change often", to "things that don't".

For dependency management, this usually means it is recommended to install dependencies in a docker layer first (as they do not change often), then install the local package(s) (as it changes at ~each build, so a lot more often).

A very simple python Dockerfile may look like

FROM python:3.12.2-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install -r requirements.txt

COPY sample.py .

where requirements are installed before copying local code, to maximise cache hit.

It is not clear how something similar could be achieved with pixi currently. A couple of options that come to mind

1. Not include local code in an environment, and use this environment to install dependencies. In that case, local code would need to be installed by something else than pixi, which does not feel ideal
2. Have two environments, one 'dependencies-only' and one 'dependencies + local code' and start by installing one, then copy the code, then install the second one. this may be better, but feels like a bit of a hack, having two environments when we actually only need one, and pixi will do extra work to solve environment.

Another alternative (and better?) option could be extra options for pixi install. For instance --no-path-dependencies and --only-path-dependencies to respectively exclude path dependencies from solve & install, or deal only with them (ideally without doing any solve work, bypassing as many steps as possible to speed things up).

The Dockerfile may then look like

FROM python:3.12.2-slim

WORKDIR /app

COPY pixi.lock pixi.toml .

RUN pixi install --locked --no-path-dependencies

COPY sample-package . 

RUN pixi install --locked --only-path-dependencies

[tdejager]

Thanks for the issue, seems like something we should facilitate @pavelzw could you chime in here with your experience?

[pavelzw]

Hmm, I get the advantage of this, although i'm worrying a bit that this makes the CLI more complicated and cluttered. I can't think of any better solution than Olivier's so I guess this could make sense in pixi...

Side note: i've not really used path dependencies because of https://github.com/prefix-dev/pixi/issues/1046 and https://github.com/prefix-dev/pixi/issues/1340#issuecomment-2107093071

---

With the postinstall method that I am using until now, we can just ship around this problem:

FROM ghcr.io/prefix-dev/pixi

WORKDIR /app
COPY pixi.lock pixi.toml .
RUN pixi install --locked
COPY pyproject.toml my_app .
RUN pixi run postinstall

But since postinstall could be considered a hack as well, this might not be what we want.

[olivier-lacroix]

Thanks @pavelzw for your thoughts on this!

As additional background, looking at other tools:

• conda install has --no-deps and --only-deps options
• pip install has --no-deps options
• poetry install has --no-root, --only-root as well as --no-directory options

conda and pip options are a bit different from what is proposed here, which can be explained by the fact they do not use lockfiles. Poetry's --no-directory option however feel very similar to what is proposed here.

[ruben-arts]

With pixi build in the back of my mind I think it would make sense to have some of these options available, even without the pypi workflow described in this Issue but including that.

I believe pixi install could be more feature full

• pixi install package_x package_y: install package_x & package_y from the pixi.lock and their dependencies.
• pixi install package_x --force-reinstall: Install the package without cache and do whatever required to install as if installing a clean machine. (helpful to avoid issues like #1695 and problems with the cache because of locally build conda packages.)
• pixi install package_x --only-deps: Don't install package_x but only it's dependencies.

I don't see why --no-deps would make sense but please enlighten me.

[olivier-lacroix]

@ruben-arts I agree on the lower usefulness of —no-deps. This tends to be done to bypass costly checks, and/or defer the installation of dependencies to a later step/stage.

[olivier-lacroix]

I am not sure using the same command pixi install for both environment level operation (as it is today) and package level is the best option; I feel it may be confusing.

Maybe a separate command for package level operation would be better ? e.g. pixi force or something like that? After all, these operations at likely outside of the « happy path ».

[gohweixun]

Hello, just checking back in to see if there have been any updates on this front. I think that separating out the dependencies installation step from the actual app installation could greatly facilitate development for containerised workflows (and this is a fairly common use case).

[olivier-lacroix]

I have created a PR that intends to solve this issue. Let me know what you guys think.

[olivier-lacroix]

Alright, so pixi install will soon have the ability to skip packages. The skipping is, for now at least, explicit:

pixi install --frozen --skip my-local-package

Will install from the lock file, except for my-local-package. Noting that

• this is not limited to local packages, and any name that matches a name in the lock file will work.
• several packages can be skipped by repeating the --skip argument