make pip install reproducible by publishing dependencies from lock file

[houbie]

Is your feature request related to a problem? Please describe.

#1883 showed that published versions of PDM (or any other python tool) can suddenly break because of a release of a dependency. Despite the usage of lock files, we publish loose dependency constraints to pypi. This makes sense for libraries, but not for standalone tools where reliability over time is important.

Describe the solution you'd like

A config option tool.pdm.publish-locked-dependencies, resulting in listing locked versions in the distributions' METADATA, would make pip installs reproducible.

[sigma67]

@frostming is this something you would accept a PR for?

[frostming]

@sigma67 certainly, appreciate that. But the locked dependencies lose the environment markers, which isn't easy to overcome.

[sigma67]

I would be willing to tackle this. Could you assign and provide some pointers for getting started?

Edit: I think it would be good idea to make this behavior optional. I.e. publish a separate locked group in optional-dependencies, allowing a pip install mypkg[locked].

[sigma67]

So I had a look at the code and came up with a basic implementation plan.

pdm

build.py:do-build()

• if project.pyproject.settings.get("publish-locked-dependencies") is set only, do the following
• modify config_settings to add an extra key locked-dependencies, which maps to a List[Dict[str,str]] of locked dependencies. Locked dependencies are retrieved from project.lockedrepository.candidate_info

add appropriate test

pdm-backend

wheel.py:_fix_dependencies

• based on self.config_settings, determine if locked_dependencies key exists
• if so, use the data within to create another optional-dependencies group named locked, which contains the same dependencies as metadata['dependencies'], but locked to the versions contained in self.config_settings['locked_dependencies']

add appropriate test

Do you agree with this proposal? Any suggestions?

[sigma67]

I was made aware that this is likely in conflict with PEP-621.

Instead we are now considering implementing a PDM plugin to build a separate, locked version of the project.

The command could look like pdm build --locked to build a modified version of mypkg, for example called mypkg-locked, which includes mypkg and all its dependencies & optional-dependencies pinned to the version specified in pdm.lock.

[frostming]

You shouldn't rely on PDM, which should be backend-agnostic. All changes should happen in pdm-backend

[sigma67]

As I mentioned above, I don't think we can achieve this in pdm-backend due to PEP621. Build backends cannot modify dependency data specified statically when creating core metadata. The only way to achieve locked dependencies then would be to mark dependencies and optional-dependencies dynamic and specify them somewhere in tool.pdm, which is probably not what we want.

[sigma67]

We have a working plugin internally, if there is any interest from the community we can work towards publishing it.

[houbie]

Super! I would use it for sure and hope that PDM itself will use it. Unless there is another way to make sure that future installs of PDM can't be broken anymore by downstream dependencies like in #1883

[sigma67]

Here it is: https://github.com/sigma67/pdm-build-locked

[houbie]

Wouldn't it make sense for PDM itself to publish releases with locked dependencies to pypi?

[sigma67]

This would be a topic for @frostming to consider. I'm not sure if the recursion is a problem

[frostming]

It's not necessary, publishing the versions from lock file is only needed if you want to publish an application (CLI or web) to pypi, and it's convenient enough to just add a pdm-build-locked dependency

[sigma67]

Exactly, pdm is a CLI application that is being published to pypi?

@houbie request is about using this plugin during the build of pdm itself if I understand correctly. Not about including the code of this plugin in pdm

[pawamoy]

PDM is both a CLI and library. Users depending on it as a library really do not want the locked dependencies. Generally speaking, any package (for example published to PyPI) should not pin dependencies, because any package (whether it advertises itself as a CLI tool, a library, an app, whatever) can be used and will be used as a library.

Let me check how pdm-build-locked works though. If it only adds a locked optional extra, this might work for PDM. If the suggestion is to publish a second pdm-locked package or something, this would work too.

[sigma67]

Let me check how pdm-build-locked works though. If it only adds a locked optional extra, this might work for PDM

Exactly, that's how it works.

We use it extensively internally for tools that target both library and CLI use cases.

The user decides how to install -

pipx install pdm[locked]

installs pinned dependencies. Nothing changes for the regular case

pipx install pdm

Existing optional dependency groups are duplicated as pinned and postfixed with -locked

[pawamoy]

OK, that's quite cool, amazing job @sigma67 :smile: I could see such a mechanism become a standard recommendation for when users install packages in isolated venvs, to be used as tools only (typically, with pipx).

[frostming]

@sigma67 can you help make the changes in a PR? Thanks