Static Build of pyo3 (embedding python interpreter)

[uncotion]

trafficstars

I would like to run python from rust, but in my situation, end users who will run the program, haven't preinstalled python and it's not possible to install (their system is extra restricted). Is it possible build a rust program that uses pyo3 will <U>embed</U> python interpreter? (So that users don't need to install a python interpreter separately)? Is it possible please give me some hit, In documentation there wasn't any point to this.

[konstin]

That's currently not supported, though it would be possible. I've written the necessary instructions for linux in the fourth of #276

[konstin]

To implement this cargo:rustc-link-search=native must be set to the value of LIBPL in the python sysconfig and cargo:rustc-link-lib=static= must be used instead of cargo:rustc-link-lib. It should be possible to enable this with a feature flag.

[hfingler]

Did anyone have any success related to this? I'm about to try this and any information would be helpful. Will report back with results.

[tr4cks]

Hello everyone,

I was also interested in this feature. I looked into it and I thought of directly integrating the embedded library provided by python rather than compiling it statically. I think it's much easier to deploy since you just have to download the python library in portable format and integrate it into your own project. I first tried to indicate the python executable via the PYTHON_SYS_EXECUTABLE environment variable but this solution doesn't work. It doesn't work because the method used to know the path of the python library is to execute a python script and get the variable returned by the function named sysconfig.get_config_var('LIBDIR'). However, in the case of the portable python library, this function returns nothing.

To do this, I thought of providing another environment variable allowing to indicate the archive containing the portable python executable. This would then be unzipped into the target/{...}/out folder and automatically linked to the project. Or much more simply leaving the possibility to overload the location of the python library. I think the latter solution would not require much change.

Tell me what you think about this?

[jmwright]

Has there been any progress on this? I see cargo:rustc-link-lib=static= in a couple of places in build.rs.

https://github.com/PyO3/pyo3/blob/master/build.rs#L333

[davidhewitt]

I investigated this briefly, but I think even with the static linking "supported" we have plenty of missing symbol errors which are seen in other tickets #763 #742 .

I believe that the problem is that unused symbols get removed at link-time, which is problematic when loading other Python extensions from a statically embedded Python interpreter.

I asked about this on URLO, but got no response: https://users.rust-lang.org/t/embed-whole-python-interpreter-using-static-linking/42509

Not sure if to get this working properly we need to add changes to cargo / rustc.

[jmwright]

@davidhewitt Thanks for the update.

[stuhood]

FTR: https://pyoxidizer.readthedocs.io/en/stable/rust.html seems to work well for embedding, although there are some limitations around which native python extension modules can be embedded along with the interpreter.

[neoblackcap]

IMHO, I think if someone want to distribue application with embedding python interpreter, he/she doesn't need to static link python interpreter with rust-base application. Write a bash script set LD_LIBRARY_PATH and name it as wrapper and distribute all shared-libraries with it just like what sublime text do.

[davidhewitt]

If you really do want to embed statically, I've managed to get a very basic POC to function on unix. It might have issues I haven't run across yet; it does successfully build (using nightly) and can import numpy.

I had to emit this slew of cargo options in pyo3's build.rs:

println!("cargo:rustc-link-arg-bins=-Wl,-Bstatic");
println!("cargo:rustc-link-arg-bins=-Wl,--whole-archive");
println!("cargo:rustc-link-arg-bins=-lpython3.9");
println!("cargo:rustc-link-arg-bins=-Wl,-Bdynamic");
println!("cargo:rustc-link-arg-bins=-Wl,--no-whole-archive");
println!("cargo:rustc-link-arg-bins=-lz");
println!("cargo:rustc-link-arg-bins=-lexpat");
println!("cargo:rustc-link-arg-bins=-lutil");
println!("cargo:rustc-link-arg-bins=-lm");
println!("cargo:rustc-link-arg-bins=-Wl,--export-dynamic");

With those in place, I could compile this code:

fn main() {
    unsafe { pyo3::ffi::Py_Main(0, std::ptr::null_mut()); }
}

using this command line:

RUSTFLAGS="-C relocation-model=dynamic-no-pic" cargo +nightly -Zextra-link-arg build

The resulting program worked like a normal python interpreter:

$ target/debug/pyo3-scratch
Python 3.9.1 (default, Dec  8 2020, 02:26:20)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.$ target/debug/pyo3-scratch
Python 3.9.1 (default, Dec  8 2020, 02:26:20)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.20.1'
>>> x = numpy.ones((100, 100))
>>> x
array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]])

So what I think about this: while we're still a long way from supporting static linking properly, this at least proves that it is possible. What would need to be improved to make this something that I'd consider ready to ship in PyO3:

• Understand if there's a better set of linker flags to generate. Can we read at least some of them from Python in PyO3's build.rs? (Maybe it's in sysconfig somewhere.)
• Those linker flags probably want to apply only to a specific bin target. Can we do that rather than unconditionally put libpython.a into all bins?
• RUSTFLAGS="-C relocation-model=dynamic-no-pic" forces full rebuilds all the time because (at least in my setup) it conflicts with the flags from rust-analyzer. Maybe this can be set through a linker flag too.
• At the moment the cargo flags require nightly Rust. I'm not totally against only supporting this on nightly Rust, but it's a little painful.
• -Wl,--export-dynamic add all symbols in the executable to the exported symbol table. This is needed to be able to import C extensions like numpy. In practice we don't need to expose all symbols - just the same set of symbols from libpython.a. Is there a different flag to use?

Hopefully the notes here serve as a useful reference for the future.

[vojtechkral]

@davidhewitt Chances are you could get (at least some of) those flags from pkg-config.
On my system:

$ pkg-config --static --libs python3-embed
-lpython3.9 -lcrypt -lpthread -ldl -lutil -lm 

I think Rust has official bindings...

[vojtechkral]

The above is a pretty minimal set of libraries needed by the Py runtime itself. Of course, there may be other libraries (such as lxml etc.) needed depending on what code exactly the user plans to run. I don't think Pyo3 itself can really tell that beforehand. The user would need to supply that info. Chances are this could be done through .cargo/config , or, if not, have a custom pyo3-specific configuration file (I don't know if there already is one).

[davidhewitt]

Thanks, interesting. You might want to see https://pyo3.rs/v0.15.1/building_and_distribution.html#advanced-config-files

[vojtechkral]

So, I had a look at how PyOxidizer do what they do. I think this is the best explanation & demo: https://pyoxidizer.readthedocs.io/en/latest/pyoxidizer_rust_generic_embedding.html#embed-python-with-pyembed

In summary:

• It's built on top of pyo3
• They have a pre-built per-OS-arch package containing statically built python and native dependencies, the CLI tool downloads this
• The CLI tool generates a bunch of files used to put the resulting binary together:
  • A config file for pyo3 build
  • An archive (custom format I think?) of Python sources and modules, presumably the standard library
  • A Rust source file to be included in main that provides configuration for the pyembed pyo3 wrapper, including the above archive as bytes ... basically this is all used to bootsrap the python interpreter and this is the purpose of the pyembed crate
  • ... some more stuff I'm not so sure about yet ...

The pyo3 config file generated on my system:

implementation=CPython
version=3.9
shared=false
abi3=false
executable=/home/vojtech/.cache/pyoxidizer/python_distributions/python.1c490d71269e/python/install/bin/python3.9
pointer_width=64
build_flags=WITH_THREAD
suppress_build_script_link_lines=true
extra_build_script_line=cargo:rustc-link-lib=static=python3
extra_build_script_line=cargo:rustc-link-search=native=/home/vojtech/scratch/pyembed/pyembedded
extra_build_script_line=cargo:rustc-link-lib=crypt
extra_build_script_line=cargo:rustc-link-lib=dl
extra_build_script_line=cargo:rustc-link-lib=m
extra_build_script_line=cargo:rustc-link-lib=pthread
extra_build_script_line=cargo:rustc-link-lib=rt
extra_build_script_line=cargo:rustc-link-lib=util
extra_build_script_line=cargo:rustc-link-lib=static=X11
extra_build_script_line=cargo:rustc-link-lib=static=Xau
extra_build_script_line=cargo:rustc-link-lib=static=bz2
extra_build_script_line=cargo:rustc-link-lib=static=crypto
extra_build_script_line=cargo:rustc-link-lib=static=db
extra_build_script_line=cargo:rustc-link-lib=static=ffi
extra_build_script_line=cargo:rustc-link-lib=static=gdbm
extra_build_script_line=cargo:rustc-link-lib=static=lzma
extra_build_script_line=cargo:rustc-link-lib=static=ncursesw
extra_build_script_line=cargo:rustc-link-lib=static=panelw
extra_build_script_line=cargo:rustc-link-lib=static=readline
extra_build_script_line=cargo:rustc-link-lib=static=sqlite3
extra_build_script_line=cargo:rustc-link-lib=static=ssl
extra_build_script_line=cargo:rustc-link-lib=static=tcl8.6
extra_build_script_line=cargo:rustc-link-lib=static=tk8.6
extra_build_script_line=cargo:rustc-link-lib=static=uuid
extra_build_script_line=cargo:rustc-link-lib=static=xcb
extra_build_script_line=cargo:rustc-link-lib=static=z
extra_build_script_line=cargo:rustc-link-search=native=/home/vojtech/.cache/pyoxidizer/python_distributions/python.1c490d71269e/python/build/lib

The lib search path contains the native dependencies, statically built:

$ ls ~/.cache/pyoxidizer/python_distributions/python.1c490d71269e/python/build/lib
libbz2.a     libformw.a        liblzma.a        libpanelw.a    libtclstub8.6.a  libXau.a
libcrypto.a  libformw_g.a      libmenuw.a       libpanelw_g.a  libtcl8.6.a      libxcb.a
libdb.a      libgdbm.a         libmenuw_g.a     libreadline.a  libtkstub8.6.a   libX11.a
libedit.a    libgdbm_compat.a  libncursesw.a    libsqlite3.a   libtk8.6.a       libz.a
libffi.a     libhistory.a      libncursesw_g.a  libssl.a       libuuid.a

It's impressive, but at the same time pretty involved and kind of opaque (for example, why use a custom archive format as opposed to eg. tar or such)...

[ghost]

@vojtechkral, How do you install python external dependencies with pyo3?

[JohnScience]

I was reading Statically embedding the Python interpreter and there I found an interesting line:

On Windows static linking is almost never done, so Python distributions don't usually include a static library. The information below applies only to UNIX.

I just checked C:\Python311\libs and I found the following dirs:

    python3.lib
    python311.lib
    _tkinter.lib

AFAIK, .lib are static libraries on Windows similarly to .a static libraries on Unix-family OSes.

[messense]

AFAIK, .lib are static libraries on Windows similarly to .a static libraries on Unix-family OSes.

They might also just be import libraries: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-creation#creating-an-import-library

[JohnScience]

I ran the following commands

C:\Python311\libs>dumpbin /ARCHIVEMEMBERS python3.lib > python3.txt

C:\Python311\libs>dumpbin /ARCHIVEMEMBERS python311.lib > python311.txt

and this is the result:

• python3.txt

• python311.txt

It looks like these are indeed import libraries for python3.dll and python311.dll, respectively.

[deadash]

I have a suggestion for how to handle static linking on Windows. We can package the python.dll file inside the executable, and intercept the loading of pyd files. We can load the Python modules from a resource embedded in the executable, or from a separate zip file. This way, we can achieve a single-file executable.

If anyone is interested, I can provide some examples to implement this solution.

[adamreichold]

@deadash Did you have a look at PyOxidizer mentioned upstream?