Linking `rules_python` libpython with PyO3

[omerbenamram]

🐞 bug report

Is this a regression?

Not AFAIK.

Description

When an interpreter is downloaded, rules_python names the affiliated libpython as libpython<version>.so.1.0 (https://github.com/bazelbuild/rules_python/blob/main/python/repositories.bzl#L206)

When used as a dependency:

    rust_shared_library(
        name = wheel_name,
        srcs = rust_srcs,
        crate_name = crate_name,
        edition = "2021",
        deps = rust_deps + [
            "@python//:libpython",
        ],
        **rust_kwargs
    )

the following linker flags will be emitted: -lpython3.9 -Lnative=bazel-out/k8-fastbuild/bin/_solib_k8/_U@python_Ux86_U64-unknown-linux-gnu_S_S_Clibpython___Ulib

However, the directory -Lnative=bazel-out/k8-fastbuild/bin/_solib_k8/_U@python_Ux86_U64-unknown-linux-gnu_S_S_Clibpython___Ulib only contains said libpython3.9.so.1.0, which isn't picked up by the linker as a valid match for -lpython3.9

-as-needed" "-Wl,-z,relro,-z,now" "-B/usr/lib/llvm-14/bin" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-ldl" "-lpthread" "-lstdc++" "-lm"
  = note: ld.lld: error: unable to find library -lpython3.9
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

Running against a modified rules_python which uses lib/libpython{python_version}.so instead of lib/libpython{python_version}.so.1.0 solves this.

cc_import(
    name = "libpython",
    hdrs = [":includes"],
    shared_library = select({{
        "@platforms//os:windows": "python3.dll",
        "@platforms//os:macos": "lib/libpython{python_version}.dylib",
        "@platforms//os:linux": "lib/libpython{python_version}.so",
    }}),
)

🔬 Minimal Reproduction

https://github.com/omerbenamram/rules_python_pyo3_repro/

🌍 Your Environment

Operating System: Ubuntu 22.04

**Output of bazel version:

Bazelisk version: v1.12.0
Build label: 5.2.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:02:26 2022 (1654617746)
Build timestamp: 1654617746
Build timestamp as int: 1654617746

Rules_python version: 0.11.0 Anything else relevant?: I was wondering why this was originally named .so.1.0?

[hrfuller]

It's a good question. It looks like the standalone interpreters do ship with a .so file as well. We probably wouldn't want to exclude it anymore https://github.com/bazelbuild/rules_python/blob/main/python/repositories.bzl#L173-L175. @f0rmiga would be the one to know better than about the nitty-gritty here.

[f0rmiga]

The .so file is a symlink to the real versioned shared object.

lrwxrwxrwx  1 1000 1000   19 Aug 18 14:51 libpython3.9.so -> libpython3.9.so.1.0
-rwxrwxr-x  1 1000 1000  37M Jan  1  2021 libpython3.9.so.1.0

I wonder if we swap the cc_import by a cc_library and include the 2 files as srcs, if it solves the problem for you.

[omerbenamram]

@f0rmiga swapping the cc_import for this

cc_library(
    name = "libpython",
    hdrs = [":includes"],
    srcs = select({{
        "@platforms//os:windows": ["python3.dll"],
        "@platforms//os:macos": ["lib/libpython{python_version}.dylib"],
        "@platforms//os:linux": ["lib/libpython{python_version}.so", "lib/libpython{python_version}.so.1.0"],
    }}),
)

works for my usecase.

If you'd like I can open a PR for this approach

[scasagrande]

My testing so far indicates that the above suggested change also resolves my pyo3 issue.

[scasagrande]

Since my post I have removed my dependency on libpython for my pyo3 target by enabling the "extension-module" and "abi3-py38" features for pyo3. I should have realized that sooner, whoops.

Anyways it now works for my situation on both native and cross compilation

[omerbenamram]

Fixed by #820