package resolve does not save packages in cache

[StefMa]

Right now resolving packages doesn't store packages in the cache.

This is in some situations problematic because "fully offline use" is not possible with the current implementation. Except of the fact that it is also desired to cache packages to avoid unnecessary API calls.

Found out via this discussion. See also this comment: https://github.com/apple/pkl/discussions/346#discussioncomment-8921705

[translatenix]

My understanding is that packages are always cached unless --no-cache is specified. Nevertheless, something like pkl project resolve --download or pkl download-package (without arguments) could certainly be useful.

[holzensp]

Instead of adding a --download argument, we're considering just always downloading (unless --no-cache is specified).

[translatenix]

Is there a benefit in keeping resolution and download decoupled? That was the idea behind pkl download-package without arguments. If resolution ever becomes non-deterministic, for example due to supporting some concept of “latest”, decoupling will be essential.

[holzensp]

We did discuss it. When using pkl project resolve --no-cache, it would not download things. That does feel less that optimally discoverable, to be sure, but our thinking so far is that scenarios where download should actively be avoided are rare enough not to create a whole separate command for the downloading. Why would you say latest would make decoupling essential? latest would only be newer-than-what-we-have-downloaded-in-cache when you resolve (not on a normal eval; otherwise latest means you can never build without an internet connection).

[translatenix]

It’s about actively avoiding resolution, not actively avoiding download. If all I want is to download, I should never have to worry about PklProject.deps.json changing as a side effect. According to the docs, the download-package command already exists, so I’m not sure what you mean by “whole separate command”.

[holzensp]

Avoiding resolution is fair enough. We were debating whether download-package should exist. Its current UI does not know about PklProject, through. This is why pkl download-package does not (yet?) work to download all project packages. Some more thinking required, but points taken.

[bioball]

Good point about pkl project resolve maybe not being deterministic in the future. But, I don't think it relates to "pkl project resolve should not populate the cache dir". Downloading dependencies is a matter of saving network I/O, and making subsequent pkl eval executions faster.

If we do want to lean on this being the method for "I want to download all my project dependencies", we can maybe add a flag for "don't update already-resolved dependencies" in the future, if "latest" becomes a thing.

BTW: "I want to download all my dependencies" can also be a shell script:

cat PklProject.deps.json \
  | jq -c -r '.resolvedDependencies[] | select(.type == "remote") | (.uri[7:] + "::sha256:" + .checksums.sha256)' \
  | xargs -I {} pkl download-package {}

[translatenix]

I guess I don’t mind if pkl project resolve also downloads packages, although I don’t really see the need either. I don’t think it saves much network I/O as packages will only be downloaded once in any case. Also, downloading as part of resolving won’t help all the other folks that didn’t run the resolve command themselves. Overall, not downloading by default feels more consistent with Pkl’s lazy attitude.

My gut feeling is that there should be an easy way to download all package dependencies without resolving. If the download-package command is here to stay, I’d expect it to be capable of this.

[bioball]

The resolve command does need to download in order to build the resolved dependency list. Since it's already downloading, it's actually a little strange that it's not writing them to the cache dir.

My gut feeling is that there should be an easy way to download all package dependencies without resolving. If the download-package command is here to stay, I’d expect it to be capable of this.

Fair; I can definitely see a mode in download-package that just downloads project dependencies. Maybe something like: pkl download-package --project=my-project/

[translatenix]

Oh I thought that pkl project resolve would just download metadata. If it has to download the packages, it should definitely store them in the cache, at least by default.