rattler icon indicating copy to clipboard operation
rattler copied to clipboard

feat: Relative path support in `LockFile`

Open hunger opened this issue 1 month ago • 11 comments

Description

Make rattler handle relative paths to pypi dependencies in the LockFile. This helps when checking in pixi.lock files that reference local python packages. Instead of ending up with a machine specific path in the lock file we have a relative path that should work for all developers.

This is the rattler part of prefix-dev/pixi#4680

The PR consists of several changes:

  • Bump the LockFile version: The relative paths will break when somebody goes back to a pixi version not supporting this
  • Enable non-pep508_extensions in the pep508_rs? crate. This is needed for it to parse relative URLs in the lock file correctly
  • Add an API to pass in the base directory used -- needed by pep508_rs to parse a relative path.
  • Keep the raw input for (some) path and urls as read from the lock file around. pep508_rs does turn everything into absolute paths right away, we need to keep that info somewhere.

hunger avatar Oct 19 '25 11:10 hunger

I do not think the failed CI job is related to my change: It's about running out of space while extracting the cache.

hunger avatar Oct 19 '25 16:10 hunger

Couple of questions for @baszalmstra and @hunger:

  • @baszalmstra This PR removes the from_str because of the relative directory requirements, any way around that you think?
  • I like that this does not introduce additional fields but keep everything in memory. Everyone agreed?
  • @hunger do we still catch parsing errors now that requires_dist is a string, @baszalmstra you reckon we do need a special type for that?
  • @hunger why do we need Verbatim in the conda part of the locking code?

tdejager avatar Oct 20 '25 11:10 tdejager

@tdejager:

  • Yes that is the troubling part, but wrapping some UrlOrPath into a Verbatim is a breaking change, too. We could in theory add FromStr back and make that use None base directory. That will of course break as soon as somebody uses a relative path into some LockFile with a parse error. That does not feel safe.
  • I was tempted to drop the bump to the file version when I saw how little the files changed. But I thing we need that version bump anyway: We do not want to break the relative paths just because somebody ran an older pixi version.
  • requires_dist is a string only in a first step. It gets processed, so it should continue to raise the parse errors. I'll double check tonight.
  • Good catch, it does not really make sense in the conda part. I'll remove that again.

hunger avatar Oct 20 '25 11:10 hunger

@baszalmstra any Idea why that gatekeeper is failing I tried to re-run it a bunch of times.

tdejager avatar Oct 20 '25 12:10 tdejager

The merge gatekeeper is just a job that fails if any other job fails. I used that to make "required" jobs based on conditions. It looks like the Rust / Linux_86_64 is failing, which is what causes Gatekeeper to fail.

The linux job seems to be running out of disk space when extracting the cache (crazy..). I just removed those caches, lets see if that helps.

baszalmstra avatar Oct 20 '25 12:10 baszalmstra

  • @baszalmstra This PR removes the from_str because of the relative directory requirements, any way around that you think?

No I think it makes total sense to add the base directory to the API.

baszalmstra avatar Oct 20 '25 13:10 baszalmstra

Is there anything I can do to push this forward? I do not think I can fix the Linux CI to not run out of space :-(

hunger avatar Oct 22 '25 07:10 hunger

What would really help is if you could try to integrate this in pixi! If we can show there that it solves the issue that would be great!

Im also working on a lock-file update that requires a lockfile bump, Id love to combine our work.

baszalmstra avatar Oct 22 '25 07:10 baszalmstra

@baszalmstra isn't it integrated here? https://github.com/prefix-dev/pixi/pull/4788

tdejager avatar Oct 22 '25 08:10 tdejager

Ah yes! I missed that!

baszalmstra avatar Oct 22 '25 08:10 baszalmstra

We are going to merge this in with the other lock-file break, we'll close this branch after, don't be suprised!

tdejager avatar Oct 28 '25 11:10 tdejager