Support `uv` as part of `rules_python`

[aignas]

trafficstars

Just creating this as a placeholder for discussions and documenting some thoughts/findings. rules_uv is doing its job well and replicating the functionality in rules_python would be duplicate effort. However, rules_uv does not support WORKSPACE installations (at least the releases don't advertise such support) and rules_python needs to support WORKSPACE installations.

As a result I see the following options:

1. Do not support uv. This has a drawback of longer compilation times with the included pip-compile tooling. uv is just a binary and could in theory be later reused for common operations with whl packages.
2. Support uv only on bzlmod by including rules_uv as a dependency. This would require us to expose rules_uv via @python_versions or something similar so that we don't have loads that load rules_uv within the main //python/*.bzl files to avoid breaking WORKSPACE users. However this would not allow us to use it for our own example testing because they should produce the same output with and without bzlmod. Since we cannot dog-food rules_uv, it makes it a hard proposition.
3. Write the WORKSPACE deps.bzl implementation for rules_uv so that we can depend on it.
4. Support uv by copying/reimplementing the uv support as a separate rule/macro whilst not relying on rules_uv. The drawback is that we would have to reimplement parts of it, but the benefit would be that the extra tests that we would need to have would be next to the implementation. We could also provide Windows support in this way as rules_uv does not support Windows yet due to relying on bash scripting for pip_compile.

[groodt]

Oh nice! I've got a POC of a basic uv integration of option 4. The logic is not complicated and it's much simpler if we only need to support blzmod. We can discuss at maintainers meeting, but thats the option I had planned anyway.

[aignas]

FYI @mark-thm, feel free to write down any thoughts that you may have :)

[mark-thm]

I don't have any particularly strong feelings here outside of liking uv as the toolchain to use to convert requirements input files to requrements txt files.

We'd be happy to take contributions that add WORKSPACE and/or Windows support. The latter will depend on rules_multitool getting Windows support (very close to landing). The bash scripts are trivial, but do let us customize the uv arguments and return to Bazel an executable, I'm not familiar with how to do that without a bash/batch script but eager to learn.

Besides that, the module is licensed pretty liberally, I don't feel any special ownership over what is otherwise a pretty trivial uv wrapper/there's really no heavy lifting to be done.

So all that said, if y'all would like to copy/replicate/do your own thing here, that's cool with me, too.

[aignas]

Thanks for the comments Mark!

[aignas]

Interesting uv features:

• astral-sh/uv#4524 for having local copies of the index and run unit tests to ensure particular output of uv pip compile.
• astral-sh/uv#4505 for having a single requirements.txt file.

I wonder how we could use some of those in the pip.parse and friends, but it would be interesting to experiment with them. Maybe using uv pip install --dry-run or something similar. Some ideas here:

Run uv pip install --dry-run to get the list of packages and versions that one would need to install. Then use that in the hub_repository as a way to construct the select statements or in pip extension evaluation to construct what the parse_requirements function is returning - i.e. requirement per platform.

What would help us drastically is something similar to pip install --dry-run --report.

[rickeylev]

Just going to repost some of the ideas/convo spawned from the initial PR so they don't get lost.

what do we call the rule that generates the lock file

Based on the discussion:

• uv_pip_compile SGTM -- mimics the underlying uv pip compile CLI
• Alternatively, we could keep the existing pip_compile name. Just add a backend="uv" attribute which, under the hood, looks up the uv toolchain instead. Or something more advanced like factoring out a common interface both a uv and pip-tools toolchain could implement (as mentioned in the PR)

should running compile be a build action or direct execution?

I'm in favor of making it a build action because its a bit more flexible. Using execution properties, we can control if its run localy, with/without sandboxing, with/without network access, etc, so it can be almost identical to a direct invocation.

We could also have a second target that handles direct invocation.

[aignas]

Before I forget, having it a build action would make it difficult to provide extra args to e.g. pass args to uv to upgrade a single package in your lockfile. If it is executed via 'run' then you get it for free - you can pass "-U requests" to bump the requests package in your lock file.

Am I missing something here? :)

[groodt]

I think we will end up with various different kinds of rules and actions tbh. Probably some will be build rules, some will be repo rules, some might be macros etc. I think it will depend on the UX and features we end up providing. I personally think uv is just an implementation detail. If folks want to grab or invoke uv directly, they can grab it in a genrule or off the toolchain. There is a runnable current_toolchain atm for example.

[aignas]

FYI, https://github.com/bazelbuild/rules_python/issues/1765 could be fixed if we start supporting the lock macro.

[aignas]

FYI, I had some issues with the auto-discovery when trying to download the dist-manifest.json from an internal mirror. The code here should ensure that the credentials from .netrc are appropriately sourced.

Current workaround is to just specify the actual urls to pull from.

[aignas]

FYI, regarding the dist-manifest.json parsing, the .sha256 URL in the manifest may not work with the bazel-downloader configuration and downloads from github being blocked. As mentioned previously, the workaround is to set the URLs manually.

To close this issue it would be great to hear about users using the lock macro. The next steps are to move all of the examples from compile_pip_requirements to lock and add a deprecation message once we get back from users who may be using this lock rule.

[AFMiziara]

#2872 allows to download uv binaries itself from private indexes, if I understood correctly, right? What about something similar to pip.parse for supporting extra indexes, is that something we can do with lock?