ABI3-like for non-`extension-module`

[VirxEC]

Description

I'd like for non-extension-module builds to be able to use newer Python interpreters, similar to how the ab3 feature set works. However, the feature description says it's "used when building Python extension modules" so it won't do anything for me.

My use case

I'd like to build an app with Rust <-> Python, but Rust first instead of Python first. I actually need Python 3.7 support, but I'd also like for users to be able to choose a newer Python version if they want. I've figured out how to set different Python interpreters by setting PATH before the interpreter is initialized, but I'd like it to be able to optionally use any newer Python version even if that means that I need every Python interpreter I just want one single binary for app user convenience.

[adamreichold]

Have you tried whether the existing abi3 features do not do what you want? I think the wording in the documentation might be misleading as the features basically control which part of CPython's API we use, whether for building an extension or for embedding an interpreter.

Or this is just about the DSO that the resulting binary tries to load being too specific? (If so, we should probably just make the existing features control the DSO name as well.)

[VirxEC]

I didn't try but with ABI3 it said "can't locate libpython3.9.so" or something

[adamreichold]

I didn't try but with ABI3 it said sone like "can't locate libpython3.9.so" or something

So I think the issue is the DSO name. Just to confirm, could you symlink the newer library to old name and check that everything else works? Thanks.

[VirxEC]

That would be funny if that works haha, I'll test it later today when I can.

[adamreichold]

Btw., I think you might already be able to change the DSO name by setting the lib_name property using PYO3_CONFIG_FILE to libpython3.so.

[VirxEC]

When changing lib_name to python3 instead of python3.9 from PYO3_PRINT_CONFIG=1 with the abi3-py37 feature enabled, I was just getting error: linking with `cc` failed: exit status: 1 and /usr/bin/ld: cannot find -lpython3: No such file or directory + collect2: error: ld returned 1 exit status.

So, I changed python3 back to python3.9, and deleted Python 3.9, and got error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory. I created a symlink from the source file /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 to /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0 and that does appear as if it worked, I have the version printing out and it says that it's running Python 3.7.

I check by deleting that symlink and making one to /usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0 instead of 3.7 and it appears to run with a newer version of Python just fine as well.

This isn't a usable solution for end users but if this gets patched then I would love to use PyO3 for more! Managing Python subprocesses and communicating via stdin/stdout is terrible.

[VirxEC]

My ideal solution would be for PyO3 to look through PATH and use the first interpreter that it finds compatible as a hint on which libpython to load. That way I can append the interpreter I want to use as the first thing to PATH then call prepare_freethreaded_python.

A possibly better solution be to prefer the same Python version as it does now, and if it can't load the file then look at the first interpreter it finds in PATH.

[adamreichold]

My ideal solution would be for PyO3 to look through PATH and use the first interpreter that it finds compatible as a hint on which libpython to load. That way I can append the interpreter I want to use as the first thing to PATH then call prepare_freethreaded_python.

A possibly better solution be to prefer the same Python version as it does now, and if it can't load the file then look at the first interpreter it finds in PATH.

I think embedding Python is not so much about finding the python binary but having the loader load the named DSO during program start-up. So this happens before our code actually runs.

I think we should try to adjust the DSO name based on the abi3 feature if possible, but that your system cannot link against libpython3.so seems to be problematic. Don't you have libpython3.so available on your system at all? Could you give the details on the distribution and how Python was installed?

As a somewhat orthogonal approach, you might want to consider using PyOxidizer to embed Python into your binary which simplifies things a lot if you do not need integration with the system-provided Python installation.

[davidhewitt]

If I recall correctly, this is a good summary of why we didn't support the abi3 feature for non-extension modules. We found that libpython3.so rarely (if ever?) exists, which prevents linking with and distributing against it. I think we hit problems in CI when we tried to use libpython3.so as a link target.

Note that even if the above linking worked, if you user tried to use an older Python's libpython you risk crashes at load time with missing symbols, which is not the greatest UX.

cc @encukou - as CPython's stable API maintainer, have you got any comment on what to do when a user wants to embed support for multiple Python 3 versions via the stable API?

[encukou]

I'm afraid I can't do much from the CPython side. CPython builds and installs libpython3.so (with -enable-shared, which you need libpython3.12.so). It would be great if distributors shipped it, like they ship the python3 command in addition to python3.12. Perhaps ask the CI provider?

[VirxEC]

Could you give the details on the distribution and how Python was installed?

I'm just using Ubuntu 22.10... it comes with a minimal install of Python 3.10 and I'm using the deadsnakes PPA to install older Python versions.

[encukou]

Pages for those are here and here, they should have all the relevant links. In the ideal world I'd go and file the issues. Unfortunately, my TODO list is too long already :(

[VirxEC]

Note that even if the above linking worked, if you user tried to use an older Python's libpython you risk crashes at load time with missing symbols, which is not the greatest UX.

Can this be avoided by linking with the oldest version you want to support? So 3.7 in my case.

[adamreichold]

Note that even if the above linking worked, if you user tried to use an older Python's libpython you risk crashes at load time with missing symbols, which is not the greatest UX.

Can this be avoided by linking with the oldest version you want to support? So 3.7 in my case.

I don't think that will give you forward compatibility because when your binary links against libpython3.7.so, it won't find and load e.g. libpython3.10.so.

[VirxEC]

Note that even if the above linking worked, if you user tried to use an older Python's libpython you risk crashes at load time with missing symbols, which is not the greatest UX.

Can this be avoided by linking with the oldest version you want to support? So 3.7 in my case.

I don't think that will give you forward compatibility because when your binary links against libpython3.7.so, it won't find and load e.g. libpython3.10.so.

Yes but it was established that deleting the file and making a sim link to another version works. Using abi3-p37 + linking with Python 3.7 should prevent missing symbols, I hope at least in theory.

[VirxEC]

I also wouldn't mind being able to link multiple interpreters, basically requiring the system compiling for production to have 3.7, 3.8, 3.9, 3.10 etc if it's a viable solution. As long as one binary can support multiple Python versions and only one is required client-side.

[Sytten]

@VirxEC Did you end up finding a solution. I have a similar issue for our software that we ship on windows, linux (aarch64, x86, armhfv7), macos. I just want to be able to use whatever interpreter is on the system if it is at least 3.7 or 3.8. I just tried on windows and I got an error that it could not find python38.dll since I have python 3.11 on the machine but I compiled against 3.8.

I had the idea of allowing the user to download python inside the app, but I still can't figure out if I am going to be able to lazy load it correctly after its been downloaded. There might be something to do with the libloading crate...

[VirxEC]

@Sytten Unfortunately the current solution is not using PyO3. I would love to but I haven't found any proper solution with it. Right now I just searches the user system for Python, and the user through setting the stuff up in the GUI (for a further example, creating the virtual environment after letting the user select the python they want to use if they have multiple.)