watchfiles 0.21.0 Configuring error

[pathway27]

Describe the issue

I seem to get this weird error when trying to use the dagster-webserver python package. Any idea on what I can do?

Additional context

% nix-shell --show-trace third_party/dag/shell.nix
these 7 derivations will be built:
  /nix/store/6njdnf1nf2z5rc2ib1hvjm22b4z7ra6h-python3.11-gql-3.5.0.drv
  /nix/store/hs6abwalf22xnn8kqpalxc9w5ix7bz00-python3.11-watchfiles-0.21.0.drv
  /nix/store/alvr3iyfbv53wa4wwcc8g65pdzzhz1si-python3.11-uvicorn-0.27.0.drv
  /nix/store/g8kyg19jb72a4y3biq2lykky5bhgzfyi-python3.11-dagster-1.6.1.drv
  /nix/store/xz9f6g1g1x8r4ci6nmryzv6hgcr9sfzg-python3.11-starlette-0.36.1.drv
  /nix/store/kszbyxkhx1bng2nr92860kl71nci7qfn-python3.11-dagster-graphql-1.6.1.drv
  /nix/store/fv1126dh0khbw5bdr9x213qg5dz56k95-python3.11-dagster-webserver-1.6.1.drv
building '/nix/store/g8kyg19jb72a4y3biq2lykky5bhgzfyi-python3.11-dagster-1.6.1.drv'...
building '/nix/store/xz9f6g1g1x8r4ci6nmryzv6hgcr9sfzg-python3.11-starlette-0.36.1.drv'...
building '/nix/store/6njdnf1nf2z5rc2ib1hvjm22b4z7ra6h-python3.11-gql-3.5.0.drv'...
building '/nix/store/hs6abwalf22xnn8kqpalxc9w5ix7bz00-python3.11-watchfiles-0.21.0.drv'...
...
Executing wheelUnpackPhase
configuring
cp: -r not specified; omitting directory '/nix/store/q3zw45mf49af911hphcpb8ng348m3qfq-source'
error: builder for '/nix/store/hs6abwalf22xnn8kqpalxc9w5ix7bz00-python3.11-watchfiles-0.21.0.drv' failed with exit code 1;
       last 10 log lines:
       > Using wheelUnpackPhase
       > Sourcing pypa-install-hook
       > Using pypaInstallPhase
       > Sourcing python-imports-check-hook.sh
       > Using pythonImportsCheckPhase
       > Sourcing python-namespaces-hook
       > Sourcing python-catch-conflicts-hook.sh
       > unpacking sources
       > Executing wheelUnpackPhase

default.nix/shell.nix/flake.nix:

{ nixpkgs ? import ../../nixpkgs.nix }:

let
  customPythonPackages = callPackage ../../tools/build/nix/python-packages {} ;
in
mkShell {
  buildInputs = [
    customPythonPackages.dagster-webserver # 1.26.0
  ];
}

pyproject.toml:

[tool.poetry]
name = "python-packages"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
packages = [{include = "python_packages"}]

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
numpy = "^1.26.0"
dagster = "^1.6.1"
dagster-webserver = "^1.6.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock:

2k lines

[torx-cz]

I have the same issue.

Is there any update or solution on how to solve it? Thank you for helping.

[iwinux]

Current workaround (as of 2024-04-21) for me:

overrides = poetry2nix.overrides.withDefaults (self: super: {
  watchfiles = super.watchfiles.override {
    preferWheel = false;
  };
});

[heimalne]

I have the same issue but for project name in my pyproject.toml file. Any new ideas why this happens?

[heimalne]

Hm, i changed a dependency in pyproject.toml, created a new poetry.lock and thereby somehow forced a new evaluation around that /nix/store/...-sources folder on the next nix-build and it worked again.

edit: The culprit dependency in our case on an M1/Arm64 Mac is this dependency:

# pyproject.toml
tensorflow-macos = {version = "2.9.0", markers = "sys_platform == \"darwin\""}

As soon as i remove this dependency at least the poetry2nix build works again.

[iwinux]

From my observation yesterday, the build failed because the hook wheelUnpackPhase was trying to unpack watchfiles, but it's not packaged as the hook expects.

[branflakes2]

Hm, i changed a dependency in pyproject.toml, created a new poetry.lock and thereby somehow forced a new evaluation around that /nix/store/...-sources folder on the next nix-build and it worked again.

edit: The culprit dependency in our case on an M1/Arm64 Mac is this dependency:

# pyproject.toml
tensorflow-macos = {version = "2.9.0", markers = "sys_platform == \"darwin\""}

As soon as i remove this dependency at least the poetry2nix build works again.

I'm seeing the same issue with tensorflow-cpu 2.14 - removing tensorflow allows my packaged to build without the cp -r ... error.

[heimalne]

Digging a bit further, the building of the project dependency (tensorflow-cpu, tensorflow-mac or watchfile) itself should work (you can check that by building the buildEnv as in poetry2nix.mkPoetryApplication.buildEnv instead).

The issue is our main project (defined by our pyproject.toml), which as a last step is turned into a wheel and then installed normally as a pyproject format installation of buildPythonPackage without the "wheelUnpackPhase". See all available formats

However, these problematic dependencies make buildPythonPackage build our main project as a "wheel" format (as if the source code was a ...arm64.whl file). I couldn't find where this logic bug occurs, but the result is that this "wheelUnpackPhase" is added through wheelUnpackHook either here

https://github.com/NixOS/nixpkgs/blob/65f3d712db1cb7227759c606c72f59c519c22413/pkgs/development/interpreters/python/mk-python-derivation.nix#L277

or here https://github.com/nix-community/poetry2nix/blob/3c92540611f42d3fb2d0d084a6c694cd6544b609/mk-poetry-dep.nix#L128

I can't quite follow the evaluation logic yet.