Installing with python bridge support

[joellama]

I’m having some issues installing GDL with the python bridge. The documentation suggests to call with —enable-shared but I can’t figure out where to add that flag. If I try:

bash scripts/build_gdl.sh configure —enable-shared I get a usage error.

Apologies if I’ve missed something in the documentation, but I couldn’t see it anywhere. Any help would be really appreciated.

[slayoo]

Hello @joellama, Are you intending to use GDL from within Python or Python from within GDL? BTW, where is it suggested to used the "--enable-shared" option? (seems an outdated resource from before we had switched to CMake). Thanks

[joellama]

Hey @slayoo, I'm hoping to use GDL from within python. The thing I'm struggling with is how to compile so that I can type import GDL within python.

The enable-shared is mentioned on line 281 of https://github.com/gnudatalanguage/gdl/blob/master/README

As a hack though I have GDL running through os.system which has enabled me to parallelize my code! I'd still love to know how to get the GDL package compiled for use though within python. Thanks again!

[slayoo]

after downloading GDL source, try something along these lines:

cd gdl
mkdir build
cd build
cmake -DPYTHON_MODULE=ON .. 
make -j
PYTHONPATH=$PYTHONPATH:`find . -name GDL.so | xargs dirname` python -m pytest ../testsuite/python/test-GDL.py 

The last line executes the Python-written tests for the Python GDL module (the one imported by import GDL) HTH

[joellama]

@slayoo, thanks for your help with this. I got a lot closer. With the commands you listed above, I now have GDL.so; however, I'm having trouble with import GDL where I get the error ImportError: dlopen(/Users/joellama/gdl/build/src/GDL.so, 2): Library not loaded: @rpath/libantlr.dylib Referenced from: /Users/joellama/gdl/build/src/GDL.so Reason: image not found

In the build/src/ directory (which is where GDL.so is), I can see antlr/libantlr.dylib but I'm not sure how to link to it. Apologies for my basic questions with this.

[slayoo]

Please try adding the directory in which libantlr.dylib is located to the list of paths in DYLD_LIBRARY_PATH or DYLD_FALLBACK_LIBRARY_PATH env vars; see man dyld.

Also, perhaps doing sudo make install will help as it should try to install the Python module within Python's site directory (https://github.com/gnudatalanguage/gdl/blob/master/src/CMakeLists.txt#L243).

HTH

[mlangelaar]

Good morning, I am having what might be a related issue ??? If you would like me to post as a new issue instead (the term hijack a thread seems to common lately, I apologize if so), then please let me know and I will do so (deleting this comment as well).

I ran into this thread after the grad student I am helping actually found it ...

EDIT -> I ran into the antlr issue weeks ago, got past that, it's just the PyInit_GDL now :( (perhaps I should post / move this to NEW issue)

1. import GDL on my system says dynamic module missing the module export function PyInit_GDL

2. I decided to use the build and test from Dec 17, 2021 that Sylwester asked the person to do. (in my case PYTHON=OFF, so I just went with what he has)

   Using the Python-written test just now, interestingly enough I am seeing an undefined symbol as shown by the test results below :

============================= test session starts ============================== platform linux -- Python 3.8.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /root/GDL/gdl plugins: filter-subpackage-0.1.1, openfiles-0.5.0, astropy-header-0.1.2, arraydiff-0.3, hypothesis-6.14.3, doctestplus-0.10.0, remotedata-0.3.2, cov-2.12.1 collected 0 items / 1 error

==================================== ERRORS ==================================== ________________ ERROR collecting testsuite/python/test-GDL.py _________________ ImportError while importing test module '/root/GDL/gdl/testsuite/python/test-GDL.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) ../testsuite/python/test-GDL.py:12: in import GDL E ImportError: /root/GDL/gdl/build/src/GDL.so: undefined symbol: _Z8wxGetAppv =========================== short test summary info ============================ ERROR ../testsuite/python/test-GDL.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! =============================== 1 error in 1.09s ===============================

Maiko

[slayoo]

CC @GillesDuvert

Seems, the compiled binary has some troubles in using wxGetApp which is part of interactive plotting / widgets machinery that is anyhow likely unusable from within the Python bridge (?). I'd suggest to try compiling without wx support by invoking cmake with

 -DWXWIDGETS=OFF -DINTERACTIVE_GRAPHICS=OFF 

[mlangelaar]

I tried that already, but I was using -DPYTHON=OFF, so I did it again just now, without that define.

This time the Python-written script is saying exactly what the import GDL is saying :

Traceback: /opt/rh/rh-python38/root/usr/lib64/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) ../testsuite/python/test-GDL.py:12: in import GDL E ImportError: dynamic module does not define module export function (PyInit_GDL)

Maiko

[slayoo]

Just to confirm, does this help: cmake -DWXWIDGETS=OFF -DINTERACTIVE_GRAPHICS=OFF -DPYTHON=OFF?

[mlangelaar]

No, searching for PyInit_GDL surprisingly (?) comes up with nothing online. I've spent hours on this, various combos with cmake, starting to wonder if this is devtool related, so I may find another server and use stock OS python and so on, the student will just have to wait ... EDIT(S) -> there was a reason to use devtools actually, stock GCC fails compile, and we needed a more recent version of Python (3), anyways ...

[slayoo]

Would it be possible for you to try the Debian package of the GDL Python module: https://packages.debian.org/sid/python3-gdl

[mlangelaar]

I can do that, will try it when I'm back at work, thank you.

On 2023-02-19 2:58 p.m., Sylwester Arabas wrote:

Would it be possible for you to try the Debian package of the GDL Python module: https://packages.debian.org/sid/python3-gdl https://packages.debian.org/sid/python3-gdl

— Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/1231#issuecomment-1436091596, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACFGMEZ5IWMYMIBT5G4XYLWYKCPTANCNFSM5KJWRTXQ. You are receiving this because you commented.Message ID: @.***>

[GillesDuvert]

Indeed, wxGetApp() is undefined since it is created in the main (gdl.cpp) program and not in a library. It is thus absent from the GDL.so The best workaround is not absolutely straightforward...

[GillesDuvert]

In other terms, as long as -DWXWIDGETS=ON Python will not work.

[GillesDuvert]

solved in #1520 pytest reports 10 failures and 49 passed tests...

[mlangelaar]

None of my cmake options are working, the GDL.so is where it should be, same thing. Gilles, do you mind giving me the cmake syntax you used for your tests ? Thanks. Maiko

[GillesDuvert]

Hi, I'm not familiar with the python 'bridge' at all. I just made sure that "wxGetApp()" was defined in GDL.so , with some caution due to the peculiar nature of the wxWidgets library.

But for the rest, I just followed @slayoo 's commands:

cd gdl
mkdir build
cd build
cmake -DPYTHON_MODULE=ON .. 
make -j
PYTHONPATH=$PYTHONPATH:`find . -name GDL.so | xargs dirname` python -m pytest ../testsuite/python/test-GDL.py 

it gave me:

$ PYTHONPATH=$PYTHONPATH:`find . -name GDL.so | xargs dirname` python -m pytest ../gdl/testsuite/python/test-GDL.py
====================================================================================================== test session starts =======================================================================================================
platform linux -- Python 3.8.14, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/testgdl
collected 59 items                                                                                                                                                                                                               

../gdl/testsuite/python/test-GDL.py ................................................xxxxxxxxx.x                                                                                                                            [100%]

================================================================================================= 49 passed, 10 xfailed in 1.07s =================================================================================================

I was not even able to find the error report that must have been created by pytest :smile:

It looks to me that no gdl executable is produced by this make, which is IMHO a bug. Besides, the location of GDL.so (or is it GDL.dyld on OSX?) seems to be different between linux and MacOS. Not mentioning all should work with a gdl installed using an installer (.exe on Windows, .dmg on MacOS) . If you confirm, then another issue should be opened.

[slayoo]

Thank you @GillesDuvert! I'm also not familiar with the Python bridge, I've never used it besides running the pytest tests that @olebole contributed.

I was not even able to find the error report that must have been created by pytest smile

Since all tests have passed, no report was issued.

[GillesDuvert]

I see 49 passed, 10 xfailed in 1.07s in the output?

[slayoo]

xfail means expected failure (e.g., a test that checks if an exception is raised for an out-of-bounds array access) - all OK!

[mlangelaar]

Quoting Gilles :

But for the rest, I just followed @slayoo 's commands:

cmake -DPYTHON_MODULE=ON .. make -j PYTHONPATH=$PYTHONPATH:find . -name GDL.so | xargs dirname python -m pytest ../testsuite/python ........ [snip]

Believe it or not, it is still failing, not able to find the PyInit_GDL

I don't want to waste your time anymore on this, it could be a variety of reasons, that I will investigate on my own time I guess. I should be clear, the minimal cmake I am using is this, since I don't have QHULL installed, and needed a self compiled version of HDF5 :

cmake3 -DHDF5DIR=/usr/local/hdf5 -DQHULL=OFF -DPYTHON_MODULE=ON ..

The only other 'questionable' thing I noticed during the cmake is this :

 -- Found wxWidgets: -pthread;;;-lwx_baseu-3.0;-lwx_gtk3u_core-3.0;-lwx_gtk3u_adv-3.0 (found version "3.0.4")
 -- INFO: We prefer to use GraphicsMagick than ImageMagick

I have installed GraphicsMagick in past attempts with no change in results.

I am using this environment, since the OS versions are 'too old' for this :

scl enable devtoolset-9 rh-python38 rh-perl530 bash

And in all fairness, some modules errored out on me during compile, to which I did a simple 'uncomment' to move forward, these were less then minor errors, suggesting that perhaps I am not meeting the proper prerequistes for packages needed to do this.

I will try (at Sylwesters suggestion) that Debian package he mentioned.

Thanks to everyone for trying, I will let you know how it goes when I have time to look at this again.

Maiko Langelaar / IT Department of Physics and Astronomy University of Manitoba

[GillesDuvert]

FYI the cmake messages above are OK, although the 'preference' to GraphicsMagick is largely a phantom of the past.

[slayoo]

Googling the "dynamic module does not define module export function" message, it seems that in some cases this is related with Python version mismatch (the one that GDL module was compiled against vs. the one that is used when importing).

In principle, there are three Python versions that must match:

• the one CMake finds with find_package(PythonInterp)
• the one CMake finds with find_package(PythonLibs)
• the one used at runtime

In the Debian Python module package source ("rules" file), they specify the following CMake options for building the module:

	  -DPYTHON=YES -DPYTHONVERSION=3 -DPYTHON_MODULE=YES

perhaps you could try with setting PYTHONVERSION=3.8.6 to make sure it is consistent?

[mlangelaar]

This is VERY helpful information, thanks Sylwester, will work with this.

On 2023-02-25 03:36, Sylwester Arabas wrote:

Googling the "dynamic module does not define module export function" message, it seems that in some cases this is related with Python version mismatch (the one that GDL module was compiled against vs. the one that is used when importing).

In principle, there are three Python versions that must match:

• the one CMake finds with find_package(PythonInterp)
• the one CMake finds with find_package(PythonLibs)
• the one used at runtime

In the Debian Python module package source ("rules" file), they specify the following CMake options for building the module:

  -DPYTHON=YES -DPYTHONVERSION=3 -DPYTHON_MODULE=YES

perhaps you could try with setting PYTHONVERSION=3.8.6 to make sure it is consistent?

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: @.***>

Links:

[1] https://github.com/gnudatalanguage/gdl/issues/1231#issuecomment-1445041298 [2] https://github.com/notifications/unsubscribe-auth/AACFGMGZMM3UPQBRZI5Z7RLWZHHDVANCNFSM5KJWRTXQ

[mlangelaar]

Adding the PYTHONVERSION actually messes up the compile believe it or not, so here's an idea now, is it possible the PyInit_GDL is a false positive ? Where does it test for this ?

Maiko

On 2023-02-25 03:36, Sylwester Arabas wrote:

Googling the "dynamic module does not define module export function" message, it seems that in some cases this is related with Python version mismatch (the one that GDL module was compiled against vs. the one that is used when importing).

In principle, there are three Python versions that must match:

• the one CMake finds with find_package(PythonInterp)
• the one CMake finds with find_package(PythonLibs)
• the one used at runtime

In the Debian Python module package source ("rules" file), they specify the following CMake options for building the module:

  -DPYTHON=YES -DPYTHONVERSION=3 -DPYTHON_MODULE=YES

perhaps you could try with setting PYTHONVERSION=3.8.6 to make sure it is consistent?

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: @.***>

Links:

[1] https://github.com/gnudatalanguage/gdl/issues/1231#issuecomment-1445041298 [2] https://github.com/notifications/unsubscribe-auth/AACFGMGZMM3UPQBRZI5Z7RLWZHHDVANCNFSM5KJWRTXQ

[slayoo]

Please provide logs from the failing compilation. I think the PyInit_GDL is called (and the error message issued) around here: https://github.com/python/cpython/blob/main/Python/importdl.c#L150-L162

[mlangelaar]

Server freshly rebooted, yum updated, latest Centos 7.9, Python 3.8.13, gcc version 9.3.1 20200408 :

mkdir build ; cd build
cmake3 -DHDF5DIR=/usr/local/hdf5 -DQHULL=OFF -DPYTHON=YES -DPYTHON_MODULE=ON ..
make -j
make install

Then try and run it as follows :

[root@ssresearch build]# python -V
Python 3.8.13

[root@ssresearch build]# python
Python 3.8.13 (default, Aug 16 2022, 12:16:29)
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/usr/local/lib/python3.8/site-packages')
>>> import GDL
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_GDL)
>>>

I have put the entire build directory on our webserver, which will include :

cmake.log, make.log, install.log, install manifest, etc ...

URL is http://www.physics.umanitoba.ca/build_ssresearch_gdl03Mar2023_maiko.tar.gz

IF you prefer I don't post a URL, please let me know ASAP and I will edit the post.

Maiko (no rush on this, I appreciate whatever you can do)