`pip_parse` produces invalid repository name for URL deps

[garymm]

🐞 pip_parse produces invalid repository name for URL deps

Affected Rule

The issue is caused by the rule: pip_parse

Is this a regression?

Yes, this works in 0.16.1 (possibly later versions too).

Description

🔬 Minimal Reproduction

git clone https://github.com/garymm/bazel-bugs.git
cd bazel-bugs/python-url-dep
bazel query @pip_pyyaml//...

🔥 Exception or Error

Error in repository_rule: invalid repository name '@pip_git+https://github_com/yaml/pyyaml_git#egg': repo names may contain only A-Z, a-z, 0-9, '-', '_', '.' and '~' and must not start with '~'

🌍 Your Environment

Operating System:

  
MacOS 13.2.1
  

Output of bazel version:

  
Build label: 6.1.1
Build target: bazel-out/darwin_arm64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Mar 15 15:45:56 2023 (1678895156)
Build timestamp: 1678895156
Build timestamp as int: 1678895156
  

Rules_python version:

  
0.20.0
  

[aignas]

I have actually checked this and it seems that there are multiple things going here:

1. If you put the git url to requirements.in like I did here then compiling the requirements succeeds and the repository name generated by pip_parse is correct.
2. If you try to use the repository, you are greeted with an error.

 (ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    pyyaml@ git+https://github.com/yaml/pyyaml.git from git+https://github.com/yaml/pyyaml.git (from -r /var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmp8ovkg2uu (line 1))
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_ignas.anikevicius/27ffab99f37f660fb65f75c56b79884b/external/python39_x86_64-apple-darwin/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/var/tmp/_bazel_ignas.anikevicius/27ffab99f37f660fb65f75c56b79884b/external/python39_x86_64-apple-darwin/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/var/tmp/_bazel_ignas.anikevicius/27ffab99f37f660fb65f75c56b79884b/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 450, in <module>
    main()
  File "/private/var/tmp/_bazel_ignas.anikevicius/27ffab99f37f660fb65f75c56b79884b/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 427, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_ignas.anikevicius/27ffab99f37f660fb65f75c56b79884b/external/python39_x86_64-apple-darwin/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_ignas.anikevicius/27ffab99f37f660fb65f75c56b79884b/external/python39_x86_64-apple-darwin/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/var/folders/nq/dvvx_cwj0d1bqv6jjntc63r40000gr/T/tmp8ovkg2uu']' returned non-zero exit status 1.
)

According to this SO question it seems that the output from pip-compile is unusable in the current form, but they may be aware of this. I am not sure how this was working in the past though.

[garymm]

Not sure if it makes a difference, but in my actual use case I specify a git commit hash in the URL, and I just updated my reproduction repo to do that as well.

[AugustKarlstedt]

+1 we're seeing this as well. the current parser doesn't seem to comply with https://pip.pypa.io/en/latest/topics/vcs-support/#vcs-support i.e. we're missing syntax support for "git ref" with an @ as well as egg and subdirectory URL fragments

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[uhlajs]

+1

[aignas]

Hey all, I understand that this is a bug report, and I would love to have some input on understanding the priority of this compared to other issues we have open at the moment.

Could someone on this issue thread explain how a fix to the parser would be beneficial and what problem it would solve in reality?

I would guess that this feature could be used for:

1. Supporting wheels with bug-fixes that have not been released yet.
2. A way to patch wheels themselves.
3. Supporting private packages that are not published to any registries.

However, since the majority code bases using rules_python I know of use wheels from private or public registries, I am struggling to understand the main use-case here. That said, if someone would like to contribute an implementation of VCS support in the starlark parser, I am happy to review and guide through the implementation.

[garymm]

My use case is number 3. Using code from a private repo without setting up a private registry.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[snakethatlovesstaticlibs]

I was able to get this to work with

some-package @ git+https://github.com/owner/package.git@<some-hash> ; python_version >= "3.x" and python_version < "3.y"

was this fixed?

[aignas]

Yeah, this way of specifying a direct dependency works, I'll close. the issue as fixed.