prefix-dev
Make global environment from pixi manifests
Problem description
Hello! Thanks a lot for the great work on pixi! I am trying to switch from micromamba to pixi and I am having some troubles making it work for our development containers.
Some context
The workflow my team has is: for each branch in a project, build a set of containers:
- a development container that contains the project's dependencies + some development tools, but not the project's source code
- a production container that contains only the project's dependencies + the installed project. Both containers are re-build regularly, once a week at least. This allows us to keep the dependencies unpinned during development: when the container are re-build we let the package manager figure out a set of working dependencies, in the meanwhile we use the ones included in the container. This way we regularly update, find and fix issues quickly, and do not develop with deprecated dependencies (which in our experience inevitably happens when someone has to manually perform the update).
I realize that our usage of containers to store and share environments overlaps with what pixi aims to achieve with its workspace design and lock files. Containers however offer great advantages when deploying applications in production, so I'd like to see if we could combine them and pixi to get the best of both worlds.
The issue
The issue comes from trying to work with the project's sources mounted from outside the container, while having the pixi environment inside the container. In the devcontainer section of the documentation, the pixi folder is mounted from the host in a volume. We would like to have the pixi environment inside the container to:
- get as close to the production setup as possible
- not rely too much on vscode devcontainer extension features. Some developers in the team use the containers directly with docker, so we keep the amount of docker options and post-entering steps to a minimum.
- share environment by sharing containers, not lock files. This way we don't have to put the lock file in source control, and can rely on the CI building the containers to regularly update the environment. I suppose we could use CI to update the lock file regularly and push it to source control, but having CI automatically push changes to repository sounds scary to me. I'd be happy to get some feedback if people are using CI like this.
I have tried the following approaches:
-
Use pixi with workspace environment (default) The issue is that the workspace is mounted from the host, so the environment is not stored in the container.
-
Use detached environments Using detached environments we can install the packages in a location in the container. The issue however is that the lock file is still in the workspace folder, so we would either have to track it with the sources, or to remove it, in which case pixi would re-download the dependencies when trying to use the environment.
-
Global environment Using a global environment seems alright when it is the only environment in a container. However, I haven't been able to make global environments from a pixi manifest.
Would it be possible to support creating global environment from pixi manifest files? I have seen the discussion about conda workflow and the proposal of the register syntax. Would it be possible to achieve the same with this feature?
Thank you!
@vincent-pollet It sounds like for your scenario, having detached environments and no lock file you would benefit from a --no-lockfile flag or, alternatively, a --no-check/--assume-installed mode that completely ignores the pixi.toml dependencies & pixi.lock.
Note: we already have --frozen and --locked flags, but they both check/rely on the lockfile.
If we had that feature, then your option 2 with detached environments would probably work quite nicely. Do I understand that correctly?
@vincent-pollet It sounds like for your scenario, having detached environments and no lock file you would benefit from a
--no-lockfileflag or, alternatively, a--no-check/--assume-installedmode that completely ignores thepixi.tomldependencies &pixi.lock.Note: we already have
--frozenand--lockedflags, but they both check/rely on the lockfile.If we had that feature, then your option 2 with detached environments would probably work quite nicely. Do I understand that correctly?
I think so. What we would need is something in between the current global and detached environments: an environment that can be created from a manifest pixi.toml (which detached env do but not global env) but that can then be used and updated independently of the manifest and lock files (which global env do, but not detached env).
Edit: to me it sounds more like a conda/mamba environment, which pixi's global environment aim to replace, that is why I was thinking to add this feature to them.
Playing with pixi a bit more, I think simply having the location of the lockfile been configurable may be enough to support this scenario?
As far as I understand, at the moment the lockfile is created by pixi next to the workspace manifest, and the hash of the lock file is stored in the environment. If the location of the lockfile could be changed to somewhere outside the workspace, then we could put them somewhere in the dev container and there wouldn't be an issue.
I don't know what would be the best way to configure this. I can see several options:
- a global option, when enabled the lockfile would be stored in the environment (next to its hash for instance). This could work for both normal or detached environments
- a global option which can be set to a directory by the users. The lockfiles of all workspaces would be stored in this directory
- an option in the pixi manifest, that users can set to indicate the location of the lockfile for only this workspace. I don't think that would be a good solution as is makes the manifest specific to the user install.
Would you consider adding this feature?
I think this is covered by https://github.com/prefix-dev/pixi/issues/4461 so closing