Dependency markers in pyproject.toml aren't used

[bouk]

Describe the issue

I have a broken test case here: https://github.com/nix-community/poetry2nix/compare/master...bouk:poetry2nix:open3d-markers

Basically, I'm trying to use system markers to install a specific version on a specific platform. These markers aren't emitted into the lockfile (any more?) by poetry so poetry2nix doesn't use them.

I guess poetry itself looks at pyproject.toml to figure out what version to install.

This seems relevant: https://github.com/python-poetry/poetry/issues/3959

Resolution

Looking into this, it seems the strategy poetry2nix currently uses is to just take all the packages in poetry.lock and install them, while poetry uses a solver to figure out what to install, even if the lock file doesn't need to change. If poetry2nix is supposed to be fully compatible then we'll have to port that to nix.

[adisbladis]

Indeed we aren't using the markers from pyproject.toml. We have all the logic implemented to do so though (markers are also used in poetry.lock where we do check them), so adding support in pyproject.toml shouldn't be too hard.

To whomever decides to tackle this: The support is implemented in pep508.nix in the function mkEvalPep508. You can look at how this is used in default.nix.

[bouk]

And note that to do it properly you need to implement the pubgrub and walk the dependency tree from the pyproject.toml to figure out what packages are actually required.

[adisbladis]

And note that to do it properly you need to implement the pubgrub and walk the dependency tree from the pyproject.toml to figure out what packages are actually required.

No, this is exactly the same pep 508 marker evaluation as we already do for poetry.lock, just in a different spot.

[bouk]

Yeah it needs to use that evaluator to determine which top-level packages need to be installed, but poetry uses the solver also during installation to figure out which packages in the lock file actually need to get installed. For example, if you have markers to select a package only on windows, then that package might also have transitive dependencies that should only get installed on windows.

[bouk]

I figured out a nice way to filter out packages on macOS, pass this as overrides to mkPoetryApplication:

overrides = [
  (import ./poetry-overrides.nix)
  pkgs.poetry2nix.defaultPoetryOverrides
  (self: super: { pyrealsense2 = if self.pkgs.stdenv.isDarwin then null else super.pyrealsense2; })
];