ModuleNotFoundError: No module named

[dprogm]

trafficstars

Hello, what could be the reason for the following error message when trying to run my py_binary: ModuleNotFoundError: No module named module_name

The shared libraries are there beside the py binary but it seems they won't be loaded properly.

[dprogm]

Can anybody help out?

[Mizux]

On which OS did you work ? Can you share a minimal example so we can reproduce it ?

note: on my side I have this example https://github.com/Mizux/bazel-pybind11

[phaedon]

I'm running into the same problem, on Ubuntu and Miniconda with a python3.8 workspace, trying to use the toy example from the tutorial at https://pybind11.readthedocs.io/en/latest/basics.html#creating-bindings-for-a-simple-function .

"bazel run pybind_example" fails with the ModuleNotFoundError reported by the original poster:

INFO: Running command line: bazel-bin/cpp-python-interface/pybind_example
Traceback (most recent call last):
  File "/home/.cache/bazel/dddb62a8db6bb2a7848883c4d384223c/execroot/__main__/bazel-out/k8-fastbuild/bin/cpp-python-interface/pybind_example.runfiles/__main__/cpp-python-interface/pybind_example.py", line 1, in <module>
    import example
ModuleNotFoundError: No module named 'example'

My cpp file looks like:

#include "cpp_example.h"
#include <pybind11/pybind11.h>

namespace pybindexample {

int add(int i, int j) { return i + j; }

} // namespace pybindexample

PYBIND11_MODULE(example, m) {
  m.doc() = "pybind11 example plugin"; // optional module docstring
  m.def("add", &pybindexample::add, "A function that adds two numbers");
}

and my BUILD.bazel file:

package(default_visibility = ["//visibility:public"])

load("@pybind11_bazel//:build_defs.bzl", "pybind_library")
load("@pybind11_bazel//:build_defs.bzl", "pybind_extension")

pybind_library(
    name = "cpp_example",
    srcs = ["cpp_example.cc"],
    hdrs = ["cpp_example.h"],
    deps = [
    ],
    alwayslink = 1,
)

pybind_extension(
    name = "pybind_extexample",
    deps = [
        ":cpp_example",
    ],
)

py_binary(
    name = "pybind_example",
    srcs = ["pybind_example.py"],
    data = [
        ":pybind_extexample.so",
    ],
    python_version = "PY3",
)

[rwgk]

I cannot help a lot, but maybe quick question will help:

• Do you see the example.so file?
• Assuming that exists, the PYTHON_PATH needs to point there.

How that's meant to work with bazel idk, sorry. All I can say is that I'm using pybind11_bazel for building and testing pybind11_abseil. It not pretty for sure, but it does work.

[phaedon]

Thanks Ralf, I do see that the shared libraries have compiled successfully -- both the ones generated by pybind_library, and the ones from pybind_extension which are supposed to be able to be imported in the data dependencies as described in https://github.com/pybind/pybind11_bazel#readme . Here's my directory listing:

(py38) LAPTOP-M4FP5N2C:~/basiccpp/cpp-python-interface$ ls ../bazel-bin/cpp-python-interface/*so*
../bazel-bin/cpp-python-interface/libcpp_example.so
../bazel-bin/cpp-python-interface/libcpp_example.so-2.params
../bazel-bin/cpp-python-interface/pybind_extexample.so
../bazel-bin/cpp-python-interface/pybind_extexample.so-2.params
../bazel-bin/cpp-python-interface/pybind_extexample.so.runfiles_manifest

As for a PYTHONPATH, my understanding is that it shouldn't be set when using Anaconda to manage python workspaces: https://stackoverflow.com/questions/17386880/does-anaconda-create-a-separate-pythonpath-variable-for-each-new-environment

[rwgk]

There is no example.so, I'm guessing you got ../bazel-bin/cpp-python-interface/pybind_extexample.so instead. What happens if you try to import that? (import pybind_extexample)

I think in the best case you get a different error, because this

PYBIND11_MODULE(example, m) {

needs to match the name of the .so.

Getting all ends to match up isn't easy. And I don't know anything about Anaconda.

When I'm in a situation like this, a lot of uncertainty and severe lack of relevant expertise, I usually try to find something that works first (e.g. I'd work with pybind11_abseil because I use it already) and then modify from there. Is there some other pybind11_bazel-based system you're using already?

[Mizux]

my 2 cents, you can take a look at: https://github.com/Mizux/bazel-pybind11 as template project with few ci jobs (e.g. https://github.com/Mizux/bazel-pybind11/actions/runs/4979586419) note: need to patch pybind11_bazel to fix windows support -_-

[rwgk]

note: need to patch pybind11_bazel to fix windows support

Please send any patches here. As long as the pybind11_abseil tests are still working (which I can test interactively pretty easily), I'll merge it. The bar is very low at the moment. It can only get better.

[Mizux]

Please take a look at (currently writing it): TLDR: Bazel 7.0 should fix an issue so we can build proper native python lib (otherwise there is our hackish patch) https://github.com/Mizux/bazel-pybind11/issues/4#issue-1736947338

[rwgk]

Please take a look at (currently writing it): TLDR: Bazel 7.0 should fix an issue so we can build proper native python lib (otherwise there is our hackish patch) Mizux/bazel-pybind11#4 (comment)

https://github.com/bazelbuild/bazel/commit/073f54b9a7c46afd2c28b4a99a235bdd6b63bb5f patches bazelbuild, there is nothing I can help with there.

note: need to patch pybind11_bazel to fix windows support

I can only merge pybind11_bazel PRs.

[jeh15]

I figured I'd post this as I was running into the same issue.

For reference this is what my example.cc (the same as the tutorial) and BUILD files look like: example.cc

#include <pybind11/pybind11.h>

namespace py = pybind11;

int add(int i, int j) {
    return i + j;
}

PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring

    m.def("add", &add, "A function that adds two numbers");
}

BUILD.bazel

load("@rules_python//python:defs.bzl", "py_binary")
load("@pybind11_bazel//:build_defs.bzl", "pybind_extension", "pybind_library")

pybind_library(
    name = "example_cc_library",
    srcs = ["example.cc"],
    alwayslink = True,
    visibility = ["//visibility:public"],
)

pybind_extension(
    name = "example",
    deps = [":example_cc_library"],
    visibility = ["//visibility:public"],
)

py_library(
    name = "example_py_library",
    data = [":example.so"],
    imports = ["."],
    visibility = ["//visibility:public"],
)

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [":example_py_library"],
    visibility = ["//visibility:public"],
)

The thing that fixed the ModuleNotFoundError for me was the imports = ["."] in the py_library.

However, I was then getting an additional error: ImportError: dynamic module does not define module export function (PyInit_example)

Adding alwayslink = True to pybind_library fixed it.

Hopefully this is helpful to others.

[hofbi]

I could solve that issue by making sure that the pybind_extension has exactly the same name as the .cpp file, e.g.

pybind_extension(
    name = "example",
    srcs = ["example.cpp"],
    visibility = ["//visibility:public"],
)

name and srcs must match here, regardless of specifying the sources directly or via deps in a pybind_library.

I also tried the alwayslnk = True and this also worked for me.

I think, it would be great to have some example code in this repo which shows the recommended way to set this up. This was already requested in https://github.com/pybind/pybind11_bazel/issues/19.

[junyer]

https://github.com/pybind/pybind11_bazel/tree/master/examples/basic now exists.