Allow specifying dependencies directly in `pip.parse`

[ouillie]

🚀 feature request

Relevant Rules

pip.parse() (not a rule, but the module extension tag)

Description

Just putting this out there to gather feedback and see if it's worth implementing myself. I use Python as a pretty minor part of my polyglot Bazel codebase, and it has a single PyPI dependency. Call me petty, but I just don't like having to put that 1 dependency in its own requirements.txt file when all my other dependencies are listed directly in MODULE.bazel.

Describe the solution you'd like

Is there any appetite for adding a new parameter to pip.parse() called requirements (as an alternative to requirements_lock) which is just a Starlark list of strings that get parsed as though they were the lines of a requirements.txt file? This would be kinda nice for really simple cases like mine, but perhaps "not recommended" for larger Python projects.

Describe alternatives you've considered

Alternatives already exist, but if the maintainers would welcome a PR to this effect, then I can draft one up.

[alexeagle]

I'd be opposed to this idea simply because Bazel claims to provide reproducible builds, and without pinning/locking requirements you could get different transitive dependencies when you rebuild at the same commit.

[ouillie]

I don't see how that's relevant. Hashes should still be locked in MODULE.bazel.lock. The only difference would be that, rather than a requirements.txt file who's only contents are e.g. numpy==1.2.3, you would instead have a Starlark list who's only contents are numpy==1.2.3. It seems to me like requirements_lock is a bit of a misnomer because requirements.txt has never been a lock file in that sense. All the locking information seems to go in MODULE.bazel.lock.

[aignas]

I think @ouillie is correct here - we can still lock the hashes in the MODULE.bazel.lock file. However, that requires the following to be in place:

• Fully stabilize #260 and make that the default (i.e. experimental_index_url is no longer experimental).
• Support downloading packages without hashes in the requirements.txt lock file to be downloaded/setup via the experimental_index_url machinery.
• Add the necessary code for the users to supply extra packages using the pypi integration code.

I am not sure if this is the same pip extension that we have right now or a different one.

[ouillie]

FWIW this is a working example of what I was thinking: https://github.com/ouillie/rules_python/commit/1da132140a91a9460d324a95452844de46e1355c

I checked the MODULE.bazel.lock file in tests/integration/pip_parse/ and it did not contain any hashes. Wishful thinking. I suppose this implements your third bullet point. Are you saying that completing the first two would enable locking for this solution automatically?

[aignas]

I think implementing the second bullet point would be sufficient for that. Then you could specify experimental_index_url = "https://pypi.org/simple" and it might work.

[keith]

one odd use case for this is it would be a nicer way to specify dependencies for things that don't support bazel but are patched into the BCR, vs adding another file