rules_py

rules_py copied to clipboard

What happened?

It looks like the generated venv isn't complete. I cannot import repo-local packages with repo-relative import mechanism.

Version

Development (host) and target OS/architectures: Ubuntu 20.04

Output of bazel --version: 5.3.2

Version of the Aspect rules, or other relevant rules from your WORKSPACE or MODULE.bazel file:

rules_py: c89d9481d52d277e12cf0c88aa8c1c72a76500c7

Language(s) and/or frameworks involved:

Python and Bazel

How to reproduce

I created a simple workspace in a fork: https://github.com/oqton/rules_py/tree/feature/import_path_bug. To reproduce just do:

cd examples/import_path_bug
bazelisk run //libs/bar

On my machine I get:

bazelisk run //libs/bar
INFO: Analyzed target //libs/bar:bar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //libs/bar:bar up-to-date:
  bazel-bin/libs/bar/bar
INFO: Elapsed time: 0.443s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
[
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python38.zip",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8/lib-dynload",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/bar.venv/lib/python3.8/site-packages",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/foo",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar"
]
Traceback (most recent call last):
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar/bar_app.py", line 6, in <module>
    from libs.foo import foo_lib
ModuleNotFoundError: No module named 'libs'

With regular Python rules running the target works as expected.

I think the following entry is missing in the path:

"/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug",

The issue is in https://github.com/aspect-build/rules_py/blob/main/py/private/venv/venv.bzl#L58. If I extend the imports_depset as:

    imports_depset = depset(
        direct = [ctx.workspace_name],
        transitive = [imports_depset],
    )

, then the provided example works as expected:

bazelisk run //libs/bar
INFO: Analyzed target //libs/bar:bar (2 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //libs/bar:bar up-to-date:
  bazel-bin/libs/bar/bar
INFO: Elapsed time: 2.668s, Critical Path: 2.28s
INFO: 3 processes: 2 internal, 1 linux-sandbox.
INFO: Build completed successfully, 3 total actions
INFO: Build completed successfully, 3 total actions
[
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python38.zip",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8/lib-dynload",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/bar.venv/lib/python3.8/site-packages",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/foo",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug"
]
print_foo

@mattem How would you like to proceed?

I'm not sure we'd want to manipulate the PYTHONPATH here, as we'd be back in the same set of issues that users experience under rules_python, this would also not work for IDEs. Perhaps we can do something similar to pips editable install and link the module into the venv?

So, the current situation is that (as explained above) we have these entries in the Python path

...
.../bar.runfiles/import_path_bug/libs/foo
.../bar.runfiles/import_path_bug/libs/bar

but the repo root is missing .../bar.runfiles/import_path_bug -- that one I added in my fix.

Going into a bit more details, before my fix:

cd examples/import_path_bug/
bazelisk run //libs/bar  # Fails.
cat bazel-bin/libs/bar/bar.venv.pth
../../../..
../../../../import_path_bug/libs/foo
../../../../import_path_bug/libs/bar

with my fix:

bazelisk run //libs/bar  # Works.
cat bazel-bin/libs/bar/bar.venv.pth
../../../..
../../../../import_path_bug/libs/foo
../../../../import_path_bug/libs/bar
../../../../import_path_bug

So, I think the paths are wired correctly via virtual env.

@mattem WDYT?