Third-party transitive dependencies seem to be packaged into wheels incorrectly

[jkaye2012]

🐞 bug report

Affected Rule

py_wheel

Is this a regression?

Doubtful

Description

When building a wheel using py_wheel, third-party pip requirements are being installed into an incorrect directory, preventing them from being properly installed/used in Python environments.

🔬 Minimal Reproduction

load("@rules_python//python:defs.bzl", "py_binary", "py_library")
load("@rules_python//python:packaging.bzl", "py_package", "py_wheel")
load("@python_deps//:requirements.bzl", "requirement")

filegroup(
    name = "srcs",
    srcs = glob(
        [
            "**/*.py",
            "*.py",
        ],
        exclude = ["tests/**"],
    ),
)

py_deps = [
    requirement("behave"),
]

py_library(
    name = "regression_harness",
    srcs = [":srcs"],
    visibility = ["//bt:__subpackages__"],
    deps = py_deps,
)

py_package(
    name = "regression_harness_pkg",
    deps = [
        ":regression_harness",
    ],
)

py_wheel(
    name = "regression_harness_wheel",
    distribution = "regression_harness",
    version = "1.0.0",
    deps = [
        ":regression_harness_pkg",
    ],
)

🔥 Exception or Error

The wheel builds as usual, and looks (at least mostly) correct. However, upon installing the wheel, we see the following:

gitpod /workspace/tribe $ ls /tmp/test/lib/python3.8/site-packages/
bin                       pip-24.1.2.virtualenv                  site-packages
bt                        pkg_resources                          _virtualenv.pth
_distutils_hack           __pycache__                            _virtualenv.py
distutils-precedence.pth  regression_harness-0.0.8940.dist-info  wheel
include                   setuptools                             wheel-0.43.0.dist-info
lib                       setuptools-57.4.0.dist-info            wheel-0.43.0.virtualenv
pip                       setuptools-57.4.0.virtualenv
pip-24.1.2.dist-info      share

regression_harness was in fact installed, and is available, but the third-party transitive dependencies are within a nested site-packages directory within the site-packages.

gitpod /workspace/tribe $ ls /tmp/test/lib/python3.8/site-packages/site-packages/
behave                  parse-1.20.2.dist-info  parse_type-0.6.2.dist-info  six-1.16.0.dist-info
behave-1.2.6.dist-info  parse.py                __pycache__                 six.py
__init__.py             parse_type              setuptools_behave.py

Of course, this means that they're not available within the environment, as that directory is not part of the PYTHONPATH.

I have tried stripping the site-packages prefix when building the wheel, but this makes the wheel file invalid as it will contain multiple .dist-info files.

Am I doing something incorrectly? This seems like a bug to me, but I can't imagine that others wouldn't have seen this.

🌍 Your Environment

Operating System:

  
Ubuntu Focal
  

Output of bazel version:

  
gitpod /workspace/tribe $ bazel version
Bazelisk version: v1.17.0
Build label: 6.2.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jun 2 16:59:58 2023 (1685725198)
Build timestamp: 1685725198
Build timestamp as int: 1685725198
  

Rules_python version:

  
0.27.1
e85ae30de33625a63eca7fc40a94fea845e641888e52f32b6beea91e8b1b2793
  

Anything else relevant?

[aignas]

The correct way to solve this could be to start using techniques outlined in https://blog.engflow.com/2024/08/20/migrating-to-bazel-modules-aka-bzlmod---repo-names-and-rules_pkg/

[jkaye2012]

Sorry, I don't see how that has anything to do with the problem at hand. This should function without bzlmod and it's not a problem with rules_pkg, but with rules_python

[aignas]

Ah sorry, I might be missing something about the original problem. I was reading this blog post and the topics seemed to be related - packaging files.

[zachcoleman]

@jkaye2012 Were you able to resolve this problem?

[jkaye2012]

Nope. For now, we're just duplicating all of the dependencies within the wheel directly. An ugly workaround that we would prefer not to do, but nothing else that we've tried has worked.

I can't shake the feeling that we're doing something wrong, but I also don't know what that would be.