pip parse fails on windows with rel path

[cgtinker]

🐞 bug report

Affected Rule

pip_parse

Is this a regression?

unknown

Description

The pip_parse example runs properly when executed from the repository on windows, linux and macOS. However, when importing the repro as http archive and running bazel run requirements.update, it fails on windows with a file not found error. On unix systems this issue does not occur.

🔬 Minimal Reproduction

Use the official pip_parse example, load the repro from http instead of using the relative path to the repro and run bazel run requirements.update on windows.

# WORKSPACE 
# load the repository from http instead of using the local repro
# local_repository(
#     name = "rules_python",
#     path = "../..",
# )

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_python",
    sha256 = "94750828b18044533e98a129003b6a68001204038dc4749f40b195b24c38f49f",
    strip_prefix = "rules_python-0.21.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.21.0/rules_python-0.21.0.tar.gz",
)

🔥 Exception or Error

FileNotFoundError: [WinError 2] The system cannot find the file specified: './requirements_lock.txt'

🌍 Your Environment

Operating System: Windows 10

Output of bazel version: 6.2.0

Rules_python version: 0.2.1

[chrislovecnm]

@cgtinker I verified it on windows. Have you tested on main? We are running windows CI, and this should fail CI.

[cgtinker]

Thanks for the fast response. Just checked, works on main!

[cgtinker]

Just realised, it still does not work. While the dependencies build when using the main branch, the requirements_lock.txt does not get updated. When the requirements_lock.txt contains valid information it "feels like it" works (thats why I missed it at first).

To reproduce:

1. delete all contents from the requirements_lock.txt
2. load from the main branch (instead of the archive)

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
    name = "rules_python",
    branch = "main",
    remote = "https://github.com/bazelbuild/rules_python",
)

[chrislovecnm]

We may be running into the same type of problem we have with bzlmod. Can I get a mini example?

[chrislovecnm]

I am getting weirdness from this as well. I am not able to update the lock file. I have tried using "requirements_windows" and the regular "requirements_lock".

I have tried

compile_pip_requirements(
    name = "requirements",
    extra_args = ["--allow-unsafe"],
    requirements_in = "requirements.in",
    requirements_txt = "//:requirements_lock.txt",
    requirements_windows = "//:requirements_windows_lock.txt",
)

and also

compile_pip_requirements(
    name = "requirements",
    extra_args = ["--allow-unsafe"],
    requirements_in = "requirements.in",
    requirements_txt = "//:requirements_lock.txt",
    requirements_windows = "//:requirements_windows_lock.txt",
)

I am not getting the same error, but the lock file is not getting updated.

I'm running bazelisk.exe --output_user_root=C:/b run --show_timestamps --verbose_failures --jobs=30 --announce_rc --experimental_repository_cache_hardlinks --disk_cache= //:requirements.update

I removed the contents of the "requirements_lock.txt" file, and running the update is not being updated. Same when I try to use "requirements_windows_lock.txt" as well.

[cgtinker]

@chrislovecnm this is the same issue I'm facing on the main branch. The error only occurs on the official release.

[ofey404]

+1 I encountered this on windows too.

[chrislovecnm]

This is working with

examples\build_file_generation

I updated the WORKSPACE file to

diff --git a/examples/build_file_generation/WORKSPACE b/examples/build_file_generation/WORKSPACE
index 7c74835..b853da4 100644
--- a/examples/build_file_generation/WORKSPACE
+++ b/examples/build_file_generation/WORKSPACE
@@ -61,9 +61,11 @@ gazelle_dependencies()
 # Our example uses `local_repository` to point to the HEAD version of rules_python.
 # Users should instead use the installation instructions from the release they use.
 # See https://github.com/bazelbuild/rules_python/releases
-local_repository(
+http_archive(
     name = "rules_python",
-    path = "../..",
+    sha256 = "84aec9e21cc56fbc7f1335035a71c850d1b9b5cc6ff497306f84cced9a769841",
+    strip_prefix = "rules_python-0.23.1",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz",
 )

And it is regenerating the windows requirements file.

I'm running:

bazelisk.exe --output_user_root=C:/b run --curses=yes --color=yes --terminal_columns=143 --show_timestamps  --announce_rc --experimental_repository_cache_hardlinks --disk_cache= requirements.update

[chrislovecnm]

It is working for me with the pip_parse example as well:

PS C:\Users\chris\Workspace\rules_python\examples\pip_parse> git diff .\WORKSPACE
diff --git a/examples/pip_parse/WORKSPACE b/examples/pip_parse/WORKSPACE
index 79aca14..b32744a 100644
--- a/examples/pip_parse/WORKSPACE
+++ b/examples/pip_parse/WORKSPACE
@@ -1,10 +1,13 @@
 workspace(name = "rules_python_pip_parse_example")

-local_repository(
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+
+http_archive(
     name = "rules_python",
-    path = "../..",
+    sha256 = "84aec9e21cc56fbc7f1335035a71c850d1b9b5cc6ff497306f84cced9a769841",
+    strip_prefix = "rules_python-0.23.1",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz",
 )
-
 load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")

 py_repositories()

 bazelisk.exe --output_user_root=C:/b run --curses=yes --color=yes --terminal_columns=143 --show_timestamps  --announce_rc --experimental_repository_cache_hardlinks --disk_cache= requirements.update

[carpenterjc]

Did this regress between these two versions? https://github.com/bazelbuild/rules_python/compare/0.18.0...0.19.0

It seems similar to: https://github.com/bazelbuild/rules_python/issues/1431

With version 19 and above I see something like this:

INFO: Running command line: bazel-bin/requirements.update.exe ./requirements.in ./requirements.txt None None None //:requirements.update --allow-unsafe --no-reuse-hashes
Updating ./requirements.txt
Traceback (most recent call last):
  File "C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_t318nvdg\runfiles\rules_python\python\pip_install\tools\dependency_resolver\dependency_resolver.py", line 139, in <module>
    if not os.path.samefile(requirements_txt, requirements_txt_tree):
  File "C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_t318nvdg\runfiles\python3_10_x86_64-pc-windows-msvc\lib\genericpath.py", line 100, in samefile
    s1 = os.stat(f1)
FileNotFoundError: [WinError 2] The system cannot find the file specified: './requirements.txt

This then appears to work in 0.26, but it doesn't and you can see what is doing with process monitor: https://learn.microsoft.com/en-us/sysinternals/downloads/procmon I have added a filter for process: python.exe path contains requirements.txt and WriteFile operation I can see that the requirements.txt file being written, but it happens inside the sandbox.

12:09:26.7976212 python.exe 20416 C:\dev\_bazel\zzfhe5cg\execroot\__main__\bazel-out\x64_windows-fastbuild\bin\requirements.update.exe.runfiles\requirements.txt WriteFile SUCCESS Offset: 0, Length: 59,532, Priority: Normal C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_lhuvah4r\runfiles\python3_10_x86_64-pc-windows-msvc/python.exe C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_lhuvah4r\runfiles\rules_python\python\pip_install\tools\dependency_resolver\dependency_resolver.py __main__/requirements.in __main__/requirements.txt None None None //:requirements.update --resolver=backtracking --allow-unsafe --generate-hashes --allow-unsafe --no-reuse-hashes

This seems to be within the temp sandbox, where as I would expect it to be written to the bazel source tree. process monitor reports the command line to be: C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_lhuvah4r\runfiles\python3_10_x86_64-pc-windows-msvc/python.exe C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_lhuvah4r\runfiles\rules_python\python\pip_install\tools\dependency_resolver\dependency_resolver.py __main__/requirements.in __main__/requirements.txt None None None //:requirements.update --resolver=backtracking --allow-unsafe --generate-hashes --allow-unsafe --no-reuse-hashes

If I capture the process create event for python I can see that the process isn't being started in the correct directory:

Parent PID:	18272
Command line:	C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\python3_10_x86_64-pc-windows-msvc/python.exe C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\rules_python\python\pip_install\tools\dependency_resolver\dependency_resolver.py __main__/requirements.in __main__/requirements.txt None None None //:requirements.update --resolver=backtracking --allow-unsafe --generate-hashes --allow-unsafe --no-reuse-hashes
Current directory:	c:\dev\_bazel\zzfhe5cg\execroot\__main__\bazel-~1\x64_wi~1\bin\requir~2.run\
Environment:	
	PYTHONPATH=C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__build;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__click;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__colorama;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__importlib_metadata;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__more_itertools;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pep517;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pip;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pip_tools;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pyproject_hooks;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__setuptools;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__tomli;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__zipp;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\rules_python;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\__main__;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__build;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__click;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__colorama;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__importlib_metadata;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__more_itertools;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pep517;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pip;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pip_tools;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__pyproject_hooks;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__setuptools;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__tomli;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\pypi__zipp;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\python3_10_x86_64-pc-windows-msvc;C:\Users\JAMESC~1\AppData\Local\Temp\Bazel.runfiles_ge8rposu\runfiles\rules_python
	RUNFILES_MANIFEST_FILE=C:/dev/_bazel/zzfhe5cg/execroot/__main__/bazel-out/x64_windows-fastbuild/bin/requirements.update.exe.runfiles/MANIFEST
	PYTHONSAFEPATH=1
	ALLUSERSPROFILE=C:\ProgramData
	APPDATA=C:\Users\jamescarpenter\AppData\Roaming
	BAZEL_SH=c:\Program Files\Git\bin\sh.exe
	BUILD_WORKING_DIRECTORY=C:/dev/zzzz_testpy
	BUILD_WORKSPACE_DIRECTORY=C:/dev/zzzz_testpy
	CHROME_CRASHPAD_PIPE_NAME=\\.\pipe\crashpad_2476_DNIMJBSNCLURGZVB
	COMMONPROGRAMFILES=C:\Program Files\Common Files
	COMMONPROGRAMFILES(X86)=C:\Program Files (x86)\Common Files
	COMMONPROGRAMW6432=C:\Program Files\Common Files
	COMSPEC=C:\WINDOWS\system32\cmd.exe

[carpenterjc]

It seems to me like this copy doesn't happen on process exit: https://github.com/bazelbuild/rules_python/blob/ee2cc930e33db358c469f3bd53bc3112de8045a2/python/pip_install/tools/dependency_resolver/dependency_resolver.py#L177

On windows resolved_requirements_file = _locate(bazel_runfiles, requirements_file) will be a full path on disk to the original file because runfiles are not symlink trees on windows by default.

This means if you calculate the path of: the file in the tree they are the same thing.

requirements_file_tree = os.path.join(workspace, requirements_file_relative)

We however calculate the relative path from the cwd and that will be the file in the runfiles tree, not the one in the workspace. I get the following values for the variables:

resolved_requirements_file C:\dev\esg-ssp\zzzz_testpy\requirements.txt
requirements_file_relative c:\dev\_bazel\zzfhe5cg\execroot\__main__\bazel-~1\x64_wi~1\bin\requir~2.run\requirements.txt
Same:(resolved_requirements_file, requirements_file_tree) True
Same:(resolved_requirements_file, requirements_file_relative) False

I believe this fix works on windows, but there is still the rel path missing for locating the original .in file:

diff --git a/python/pip_install/tools/dependency_resolver/dependency_resolver.py b/python/pip_install/tools/dependency_resolver/dependency_resolver.py
index e277cf9..8e8940a 100644
--- a/python/pip_install/tools/dependency_resolver/dependency_resolver.py
+++ b/python/pip_install/tools/dependency_resolver/dependency_resolver.py
@@ -170,10 +170,10 @@ if __name__ == "__main__":
             # In most cases, requirements_file will be a symlink to the real file in the source tree.
             # If symlinks are not enabled (e.g. on Windows), then requirements_file will be a copy,
             # and we should copy the updated requirements back to the source tree.
-            if not os.path.samefile(resolved_requirements_file, requirements_file_tree):
+            if not os.path.samefile(requirements_file_relative, requirements_file_tree):
                 atexit.register(
                     lambda: shutil.copy(
-                        resolved_requirements_file, requirements_file_tree
+                        requirements_file_relative, requirements_file_tree
                     )
                 )
         cli()

I think it needs a the path normalised as well: sys.argv.append(os.path.normpath(requirements_file_relative if UPDATE else requirements_out))

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"