Importing a local module

[iddan]

After running the basic example I tried to import a local Python module in my Rust project and failed. My file structure is:

| module
|-- __init__.py
| src
|-- main.rs
| target
|-- Cargo.toml
|-- Cargo.lock

Where:

__init__.py

def speak() -> None:
    print("Hello, World!")

src/main.rs

extern crate cpython;

fn hello(py: cpython::Python) -> cpython::PyResult<()> {
    let module = py.import("module")?;
    module.call(py, "speak", cpython::NoArgs, None)?;
    Ok(())
}

fn main() {
    let gil = cpython::Python::acquire_gil();
    hello(gil.python()).unwrap();
}

It fails with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: PyErr { ptype: <class 'ModuleNotFoundError'>, pvalue: Some(ModuleNotFoundError("No module named 'module'",)), ptraceback: None }', libcore/result.rs:945:5

[ssokolow]

As I remember, when embedding CPython, it doesn't put . on the import path the way it would if you were running the standalone Python runtime.

Import sys and Debug-print the contents of sys.path to verify that hypothesis. If that's the case, add an appropriate entry to the list before attempting your import and it'll resolve the issue.

(You might also want to put a test.py in the root of your project which prints the contents of sys.path to see if there are any other discrepancies that might trip you up.)

[iddan]

Output of sys.path is:

['/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/iddan/Library/Python/3.6/lib/python/site-packages', '/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages']

[iddan]

I think the t should by default include the path of Cargo.toml directory or at least provide an easy way to do so and document it.

[ssokolow]

My understanding is that:

1. CPython itself defaults that way for security and predictability reasons. (ie. Suppose a program embeds all non-library code in the compiled host executable and execs it. You wouldn't want it to fail or be exploitable because someone dropped an unexpected file into the local directory with the same name as a standard library module.)

2. rust-cpython, for security and safety reasons, tries to provide the least surprising balance of wrapping what CPython does and what is idiomatic for Rust.

Also, you can't include the path of the Cargo.toml directory because that's a compile-time thing while Python's import is a runtime thing and the Rust binary you build has no idea where that is at runtime or if it even exists on the same machine. (The closest you can get would be to have it follow CPython's behaviour, which would have it looking in <PROJECT_DIR>/target/debug or <PROJECT_DIR>/target/release or whatever non-default location you've specified by setting CARGO_TARGET_DIR)

[iddan]

How would you suggest consuming user python code safely and Rusty?

[ssokolow]

I haven't thought about it much, since I follow the Python maxim of extend, don't embed and use rust-cpython to build importable modules that setuptools-rust builds and copies into place when I run setup.py build or setup.py develop.

However, you could probably use a build script to copy the Python stuff into target/<whatever>/ and then amend sys.path to restore the CPython behaviour of searching relative to the file you launched.

[ssokolow]

Now, if you're willing to "extend, don't embed", then I can certainly recommend a way to support user-provided code.

Use a plugin framework like YAPSY. It'll...

1. Allow you to specify a plugin import path separate from the Python import path, which makes it easy to specify something like ~/.local/share/my_app/plugins as part of the import path.
2. Support a metadata file alongside the plugin code, so a listing of plugins can be displayed without having to run them first.
3. Prevent you from having to reinvent "walk the filesystem to find Python code, then import it"
4. Offer optional extras, such as ready-made code to support remembering which plugins are enabled and disabled across restarts.

YAPSY's docs are a bit spartan, so you'll want to look at the example code linked to, but I like it.

If you don't like YAPSY, there are various others available, or you could write your own.

There isn't really any way to make consuming untrusted Python code safe, since Python wasn't designed with that in mind so, if that's your goal, you probably want one of the two embeddable Lua bindings for Rust instead... or you could use Neon to extend Node.js and rely on one of the options for running untrusted JavaScript within a Node.js application but apparently a lot of them haven't been updated to account for holes that were discovered. (That's why I prefer Lua for untrusted scripting but, if you do want to go with Node.js, vm2 is the only choice for which I was unable to find any warnings about security flaws.)

[iddan]

My use case is consuming in-company Python code in our Rust web server. If I understand correctly your solution makes Python the main caller. I am willing to maintain Rust as the program entry point. How about include_dir()!? Would it be a safe option?

[ssokolow]

Python's import won't know about it, so it's really only a shorthand for a bunch of include_str! calls and you still have to build an alternative to import.

(And it's a compile-time thing, so you'll have to recompile your Rust code every time you change your Python code in order to embed the updates.)

Here's a response I gave to someone else which you may find useful for replicating import.

[ssokolow]

Oh, why do you want to maintain Rust as the entry point? I might be able to suggest an alternative.

(eg. If it's because you want the main loop to be in Rust, you could take inspiration from things like PyGTK and PyQt, where the Python code registers callbacks and then hands off control to a compiled main loop.)

[b]EDIT:[/b] Also, whether embedding or extending, rust-cpython can't magically eliminate the Python GIL. All your Python code will be serialized unless you use an approach similar to Python's multiprocessing module and split off worker subprocesses.

[iddan]

Why wouldn't I? If my main application logic sits in Rust code I'd like it to be the source of truth for my application. I can see how the connection between Python & Rust can be maintained on the build toolchain level but I think it adds an avoidable complexity. It is a very simple common use case to just want to "import" foreign lang code into another.

[ssokolow]

Because Python is friendlier to extending-based architectures than embedding-based ones.

This page, which I linked earlier, goes into more detail:

https://twistedmatrix.com/users/glyph/rant/extendit.html

The TL;DR: is that you have to reinvent fewer wheels if you put Python at the top of the call stack.