Can't package applications using modules

[Clockwork-Muse]

Although the module runs if run via python, it fails when attempted to run via packaging:

Traceback (most recent call last):
  File "__main__", line 1, in <module>
  File "importlib._bootstrap", line 991, in _find_and_load
  File "importlib._bootstrap", line 975, in _find_and_load_unlocked
  File "importlib._bootstrap", line 671, in _load_unlocked
  File "importlib._bootstrap", line 827, in exec_module
  File "panda3d.core", line 1, in <module>
  File "imp", line 342, in load_dynamic
  File "importlib._bootstrap", line 702, in _load
  File "importlib._bootstrap", line 657, in _load_unlocked
  File "importlib._bootstrap", line 556, in module_from_spec
  File "importlib._bootstrap_external", line 1166, in create_module
  File "importlib._bootstrap", line 219, in _call_with_frames_removed
ImportError: /proj/build/manylinux2010_x86_64/libpandaexpress.so.1.10: undefined symbol: _ZNK11GlobPattern14matches_substrEN9__gnu_cxx17__normal_iteratorIPKcSsEES4_S4_S4_

Repro steps:

- docker run --rm -it --entrypoint /bin/bash ubuntu
- apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install cmake curl git python3-venv python3-pip 
- curl https://www.panda3d.org/download/panda3d-1.10.10/panda3d1.10_1.10.10~focal_amd64.deb -o panda3d-1.10.10/panda3d1.10_1.10.10~focal_amd64.deb
- apt-get -y install panda3d-1.10.10/panda3d1.10_1.10.10~focal_amd64.deb
- git clone https://github.com/tobspr/P3DModuleBuilder.git /proj/P3DModuleBuilder
- cd /proj/P3DModuleBuilder
- rm -rf source/*
- cat << EOF > source/example.h 
#ifndef EXAMPLE_H
#define EXAMPLE_H
#include "pandabase.h"
BEGIN_PUBLISH // This exposes all functions in this block to python
inline int multiply(int a, int b) {
    return a * b;
}
END_PUBLISH
class ExampleClass {
    PUBLISHED: // Exposes all functions in this scope, use instead of "public:"
        inline int get_answer() {
            return 42;
        };
};
#endif EXAMPLE_H
EOF
- echo -e "\nmodule_name=TestModule\n" >> config.ini
- python3 build.py
- cd ../
- cat << EOF > s.py
import panda3d.core  # Make sure you import this first before importing your module
from P3DModuleBuilder import TestModule
print(TestModule.multiply(3, 4)) # prints 12
example = TestModule.ExampleClass()
print(example.get_answer()) # prints 42
EOF
- python3 s.py
- echo -e "panda3d" >> requirements.txt
- cat << EOF > setup.py
import setuptools
setuptools.setup(
    name="repro",
    version="0.1",
    options={
        'build_apps': {
            'platforms': [ "manylinux2010_x86_64" ],
            'console_apps': {
                'repro': "s.py",
            }
        }
    }
)
EOF
- python3 setup.py build_apps
- ./build/manylinux2010_x86_64/repro

[rdb]

I think you need to change the _GLIBCXX_USE_CXX11_ABI setting.

[Clockwork-Muse]

.... where, and how? When I build the SDK/Wheel? When I compile the module (except the module builder doesn't supply a way to pass more args)? When I build the application for distribution (see previous)? And what should it be set to?

[Clockwork-Muse]

Side note: because I'm building my own copy of the sdk/wheels, I can get it to work in a trivial repo with an ubuntu-built wheel (linux_x86_64 vs manylinux2014_x86_64, although I'm currently having issues with my real repository). That's maybe not an option for everyone though (and one I'd like to avoid as well, but panda would need to put out nightly builds so I could pick up occasional bug fixes).

[Clockwork-Muse]

... and it's not an option for me either - I'm of course using packages that only have manylinux wheels built (which is why I went for it in the first place).

[rdb]

Your repro case gives me this error:

ImportError: /proj/build/manylinux2010_x86_64/P3DModuleBuilder.TestModule.so: undefined symbol: _ZNK20ExecutionEnvironment27ns_has_environment_variableERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

The module apparently requires the cxx11 ABI version of the symbol:

# readelf -sW P3DModuleBuilder.TestModule.so  | grep ns_has_environment_variable
    23: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND _ZNK20ExecutionEnvironment27ns_has_environment_variableERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

Which is not being provided by the copy of the Panda3D libraries that ship with the manylinux2010 wheel:

# readelf -sW libp3dtool.so.1.10  | grep ns_has_environment_variable
   233: 00000000000164e0   241 FUNC    GLOBAL DEFAULT    8 _ZNK20ExecutionEnvironment27ns_has_environment_variableERKSs

So if you add this to the CMake file, telling it not to use the cxx11 and cxx17 ABIs:

  add_definitions("-D_GLIBCXX_USE_CXX11_ABI=0")
  add_definitions("-D_GLIBCXX_USE_CXX17_ABI=0")

Then it will stop being able to run with the Ubuntu Focal version of the Panda3D libraries:

# python3 s.py
Traceback (most recent call last):
  File "s.py", line 2, in <module>
    from P3DModuleBuilder import TestModule
ImportError: /proj/P3DModuleBuilder/TestModule.so: undefined symbol: _ZNK8Filename14to_os_specificEv

But it will now work correctly with the manylinux2010 version of the Panda3D libraries:

# build/manylinux2010_x86_64/repro 
12
42

The best way to ensure that your module will work with the manylinux2010 version of Panda3D is to build your own module inside a manylinux2010 container as well, but the above workaround solves your immediate problem.

[Clockwork-Muse]

The best way to ensure that your module will work with the manylinux2010 version of Panda3D is to build your own module inside a manylinux2010 container as well, but the above workaround solves your immediate problem.

I thought that might be the case, and was tweaking my build scripts to be able to do that (although I've been using 2014 so far, and presumably that will work). Thanks you.

[Clockwork-Muse]

Further update.

Making an installer in the manylinux (2014, in my case) images doesn't work, because it's missing a few packages - one of which is python, which might be a different version than panda was built against. Curing this still runs into a problem because for some reason installing the resulting .rpm doesn't result in it being recognized by the build scripts... Installing "manually" by calling makepanda/installpanda.py + ldconfig gets it recognized, but building fails during linking:

/opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: /opt/python/cp38-cp38/lib/libpython3.8.a(abstract.o): relocation R_X86_64_32 against `.data' can not be used when making a shared object; recompile with -fPIC

fulllog.txt

... which I'm taking to mean that I'd have to recompile python, which would push things farther from the original build scripts. And also means that deployable applications are farther out of reach.

What now? I'm not well versed in the arcane side of C++/OS binaries, so I'm afraid I'm a little lost here. Is it even possible to build the module in such a situation?

[Clockwork-Muse]

Note that the reason I'm doing this is that I'm trying to create my own subclasses of VirtualFileSystem/VirtualFile (to create a new remote client for a different backend). I tried doing it in python first, but of course that didn't work because it wasn't recognized from the C++ side.

[rdb]

P3DModuleBuilder is not supposed to link against libpython. It is not needed and causes problems such as the one you are having.

I think P3DModuleBuilder should be changed not to use the deprecated PythonLibs and rather use this: https://cmake.org/cmake/help/latest/module/FindPython3.html#module:FindPython3 With exclusively the "Module" target and not "Embed".

[Clockwork-Muse]

... okay, so, part of the problems being encountered is because P3DModuleBuilder needs the headers to compile against. Would including the headers in the wheels be sufficient to be able to compile modules (although I'm expecting no, I suppose)?

[rdb]

I don't understand, which headers do you need? Are you running into a new problem, different from the Python one?

[Clockwork-Muse]

Ah, no (not yet, at least). P3DModuleBuilder claims that it needs the SDK, as opposed to the wheel, because the wheel doesn't include any of the headers. If the wheel did include the headers, would it be possible to compile against only the wheel on arbitrary platforms? So I wouldn't need to build the module on a manylinux image, but could do it on my normal (ubuntu) dev/build image as a normal build step.

[Clockwork-Muse]

... strangely, your workaround actually appears to work when both using the python interpreter, and when having a built app, at least for my custom build. While the situation is still not ideal, it may work for my situation.

[el-dee]

FWIW I'm successfully using (a modified version of) P3DModuleBuilder on Linux, Manylinux, macOS and Windows to generate modules for my app.

One of the modifications I had to make was to remove ${PYTHON_LIBRARIES} from the target_link_libraries on Linux. On macOS and Windows one must still link against the Python shared library.

Also, but it might not be related to your problem, I had to set'-Wl,--as-needed linker flag on Linux. (Sadly I don't remember what was the root cause)

[Clockwork-Muse]

After having worked on this some more:

• The workaround doesn't play nicely with some of my dependencies (which are installed/managed with vcpkg)
• I can't get a working CMakeLists.txt inside a manylinux image - it complains about missing symbols (error: PyUnicode_FromString was not declared in this scope, for example).

At this point I'm probably looking at forking panda itself, which I was hoping to avoid.

[Clockwork-Muse]

MUWAHAHAHAHA! I got it working!

The big thing is that additional_libraries.cmake can't be used - or at least not without a lot more work. It works far better if you modify the CMakeLists.txt directly.

Here's the modified file that I'm using to include the gcloud storage; note that it amounts to two things:

1. The use of FindPython3, as suggested, and using the variables it finds there. Note that I'm hinting by creating and activating a venv.
2. The inclusion of the gcloud library. This was installed via vcpkg, so I needed to set CMAKE_TOOLCHAIN_FILE as necessary.

The module was built on a manylinux image, but runs on my ubuntu machine, and can contact my gcloud storage, so looks like I'm all set. CMakeLists.txt .

[Clockwork-Muse]

.... aaannnnddd it's broken again. I'm not sure what I was doing to get it to work the first time, but I've gotten another workaround going.

It only appears to be "missing" two files:

• libp3direct.so, which can be included in a build by adding "p3direct" to the plugins section of setup.py
• libp3framework.so, which cannot. This .so file, unlike the rest (found in site_packages/panda3d), is found in site_packages/panda3d_tools. As long as this file ends up in the build directory the build module works, although it complains during the build process.

@rdb - should libp3framework.so be in the same directory in the wheel as the other .sos?
I'm using a wheel built essentially the same way as the official ones, and a centos sdk that has the python include/lib directories set on makepanda.py (otherwise I can't get modules to compile on the manylinux image - it doesn't find any of the necessary python definitions in the generated interrogate files).

[el-dee]

For info, In my own fork of p3d-module-builder, I removed those two libraries from the CMakeList.txt to make the build successful and the module usable. Unless you are doing some very specific code, I don't think you need them either.

[Clockwork-Muse]

@el-dee - Yes, that indeed did it, thank you!

[rdb]

I removed p3framework in 2e8789bc14ec32eb2f64f3e669e7b4144c9bd731; it isn't really useful to extension modules, it's just for standalone C++ programs.

p3direct, on the other hand, may be useful, since it contains the interval module.