ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

[tcztzy]

🐛 Bug Reports

I am working for writing Python extension, but facing import submodule error.

ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

What do I expected

Import submodule like usual Python package.

🌍 Environment

• Your operating system and version: ArchWSL version 19.11.16.0 on Windows Subsystem Linux 2 version Windows Insiders Slow ring 10.0.19041

$ uname -a
Linux xxxxxx 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 GNU/Linux

• Your python version:

$ python -V
Python 3.8.1

• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?:

$ pyenv virtualenv system xxxxxx

• Your rust version (rustc --version):

rustc 1.43.0-nightly (c9290dcee 2020-02-04)

• Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")? Yes.

💥 Reproducing

Please provide a minimal working example. This means both the rust code and the python.

Please also write what exact flags are required to reproduce your results. src/lib.rs

use pyo3::prelude::*;
use pyo3::{wrap_pyfunction, wrap_pymodule};
use pyo3::types::IntoPyDict;

#[pyfunction]
fn subfunction() -> String {
    "Subfunction".to_string()
}

#[pymodule]
fn submodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pyfunction!(subfunction))?;
    Ok(())
}

#[pymodule]
fn supermodule(_py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

Cargo.toml

[package]
name = "supermodule"
version = "0.1.0"
authors = ["Tang Ziya <[email protected]>"]
edition = "2018"

[package.metadata.maturin]
classifier = [
    "Development Status :: 3 - Alpha",
    "Programming Language :: Python",
    "Programming Language :: Rust",
]

[lib]
name = "supermodule"
crate-type = ["cdylib", "rlib"]

[features]
default = []

[dependencies]
pyo3 = { git = "https://github.com/PyO3/pyo3", features = ["extension-module"] }

[dev-dependencies]

$ maturin develop
$ python -c "from supermodule.submodule import subfunction"
ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package

[davidhewitt]

This is unfortunately a flaw in the way Python imports modules from native extensions. I think if you run import supermodule first, then from supermodule.submodule import subfunction will succeed.

[waymost]

I've run into this same issue with version 0.8.5 on Mac and no combination of imports that I've tried has worked, including the above suggestion.

[davidhewitt]

Thanks for the note. I must have mis-remembered that.

Based on stack overflow the solution looks like you might be able to create a symlink to the main library which has the name of the submodule: https://stackoverflow.com/questions/48706842/python-3-x-c-extension-module-and-submodule

[sunjay]

You can fix this in a somewhat hacky way by adding the module to sys.modules manually. From your Rust code, this can look like:

#[pymodule]
fn supermodule(py: Python, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.run("\
import sys
sys.modules['supermodule.submodule'] = submodule
    ", None, Some(module.dict()))?;

    Ok(())
}

The documentation claims that sys.modules is writable, so this approach should not break anytime soon. The only caveat is that while this works when executed by a Python interpreter, it does not work with static analysis tools like pylint.

After a lot of trial and error, I am fairly convinced that the only "fully robust" solution that will also make static analysis tools happy is to have your entire module hierarchy represented in the file system. Whether you do this with Python files or with separate extension modules is up to you. I agree with the comments in PyO3/maturin#266 that we should look into how numpy, pandas, etc. handle this and see if there is a better way.

It would be great if in the meantime PyO3 could emit code to modify sys.modules automatically so from supermodule.submodule import ... would work out of the box.

[davidhewitt]

A solution for this came up in https://github.com/PyO3/pyo3/issues/1517#issuecomment-808664021

We could try to support this pattern internally in PyO3 (or at least add documentation).

[chris-laplante]

Somewhat unrelated but maybe this will help someone. In my particular case I had a module on-disk with a number of submodules. I only wanted to import a subset of the submodules. Tried many solutions (e.g. PyModule::from_code, py_run! workarounds, etc) but this is what finally worked for me:

let sys: &PyModule = py.import("sys").unwrap();
let syspath: &PyList = sys.getattr("path").unwrap().try_into().unwrap();

syspath
    .insert(0, "/path/to/python/library/")
    .unwrap();

let my_module = PyModule::new(py, "my_module")?;
let my_mod_submodule_1 = py.import("my_module.submodule1").unwrap();
let my_mod_submodule_2 = py.import("my_module.submodule2").unwrap();
my_module.setattr("submodule1", my_mod_submodule_1)?;
my_module.setattr("submodule2", my_mod_submodule_2)?;

[vxgmichel]

For the record, an alternative to the py.run! solution is:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;

    Ok(())
}

[davidhewitt]

NB it looks like submodules may also have complexity with things like pickling: https://github.com/PyO3/pyo3/issues/2469#issuecomment-1169570706

[hameer-spire]

For the record, an alternative to the py.run! solution is:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;

    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;

    Ok(())
}

The modern incantation to do this is something like:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let submodule = pyo3::wrap_pymodule!(submodule);
    m.add_wrapped(submodule)?;
    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule(py))?;
    Ok(())
}

[jdiggans-twist]

I'm encountering this same issue and when using the solution suggested in https://github.com/PyO3/pyo3/issues/759#issuecomment-1208179322 (via copy/paste).

After maturin develop and attempting import supermodule.submodule in a REPL I'm now getting:

thread '<unnamed>' panicked at 'failed to wrap pymodule: PyErr { type: <class 'ImportError'>, value: ImportError('PyO3 modules may only be initialized once per interpreter process'), traceback: None }', supermodule/src/lib.rs:432:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python3.9/site-packages/supermodule/__init__.py", line 1, in <module>
    from .supermodule import *
pyo3_runtime.PanicException: failed to wrap pymodule: PyErr { type: <class 'ImportError'>, value: ImportError('PyO3 modules may only be initialized once per interpreter process'), traceback: None }

I don't see any issues in this repo on this particular error (multiple initialization) - any idea where I've gone wrong?

(I see from https://github.com/PyO3/pyo3/blob/7bdc504252a2f972ba3490c44249b202a4ce6180/guide/src/migration.md#each-pymodule-can-now-only-be-initialized-once-per-process that this is new behavior as of the 28 August release of 0.17 but it's not clear from that writeup what 'the ability to initialize a #[pymodule] more than once in the same process' means - i.e. I am just starting a REPL and importing the package one time to trigger the error).

[jdiggans-twist]

If I back down to pyo3 v0.16.6, the import works without error so it's something new to 0.17.x.

[davidhewitt]

@jdiggans-twist I've created a separate issue #2644 to discuss.

[ariebovenberg]

One addition: the __name__ of the submodules isn't set correctly either.

Here is the complete solution I use for this:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let submodule = create_my_submodule()?;  // function that creates the &PyModule
    m.add_submodule(submodule)?;
    py.import("sys")?
        .getattr("modules")?
        .set_item("supermodule.submodule", submodule)?;
    // needs to be set *after* `add_submodule()`
    submodule.setattr("__name__", "supermodule.submodule")?;
    Ok(())
}

[Anexen]

Here is my solution that correctly sets __name__ for both submodule and the function:

#[pymodule]
fn supermodule(py: Python, m: &PyModule) -> PyResult<()> {
    let child_module = PyModule::new(py, "supermodule.submodule")?;
    submodule(py, child_module)?;
    m.add("submodule", child_module)?;
    py.import("sys")?.getattr("modules")?.set_item("supermodule.submodule", child_module)?;
    Ok(())
}

[nleroy917]

Hi guys,

I was able to use all the above to get submodules going for my package. Thank you! Something I couldn't figure out was adding type stubs to the package after this. I followed the tutorial, but it seems highly specific to modules that don't have submodules. Is there a way to add type's to projects that include submodules without creating the python source folder?

Thanks!

[davidhewitt]

@nleroy917 maybe https://github.com/PyO3/pyo3/discussions/3591#discussioncomment-7647350 helps you?

[nleroy917]

@davidhewitt Thank you for pointing me there, that seems to be the exact problem I was having. In the meantime, I got something working after reading this in the documentation:

As we now specify our own package content, we have to provide the init.py file, so the folder is treated as a package and we can import things from it. We can always use the same content that Maturin creates for us if we do not specify a Python source folder. For PyO3 bindings it would be:

from .my_project import *

It's not perfect, and I am repeating myself a lot, but it works and is sufficient for now. I sincerely appreciate your work on this project! It's become an indispensable tool in my workflow