Python wrapper feature: unable to import pysu2 by Python on OSX

[yiren6]

Describe the bug Python cannot import pysu2 module with error message:

import pysu2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/.../feature_pysu2_fsi_adjoint/SU2_PY/pySU2/pysu2.py", line 17, in <module>
    import _pysu2
ModuleNotFoundError: No module named '_pysu2'

This issue seems to be persistent for different OSX machines under standard build when enabling python wrapper. Tried to build wrapper through swig manually and yet this problem persists.

Desktop (please complete the following information):

• OS: Mac OSX 12.3
• C++ compiler and version: apple clang 13.1.6
• MPI implementation and version: mpich 4.0.1
• SU2 Version: 7.3
• Python Version: 3.9.7
• SWIG Version: 4.0.2

[pcarruscag]

Are you setting the paths? If you do just import SU2 does it work?

[patelha57]

Are you setting the paths? If you do just import SU2 does it work?

I have the same issue as Yiren on my Mac (with a nearly identical environment setup in terms of Python version, SWIG version, etc.). There are no issues with import SU2, only import pysu2 and import pysu2ad.

[pcarruscag]

Is that problem new @patelha57? Have you tried compiling with the bare minimum? No cgns, tecplot, any external library basically.

[yiren6]

@pcarruscag Hi Pedro. After a while we think the problem lies in the .dylib shared library file suffix. After manually changing suffix into _pysu2.so one can import/load the python module. We also tried to change meson.build file under SU2_PY/pysu2 to enforce output .so file:

      swig_lib = shared_library(
      '_pysu2',
      cpp_source,
      dependencies: [wrapper_deps, common_dep, su2_deps],
      objects: [
        su2_cfd_lib.extract_all_objects(),
        su2_def_lib.extract_objects('drivers/CDeformationDriver.cpp')
      ],
      install: true,
      include_directories : mpi4py_include,
      cpp_args : [default_warning_flags,su2_cpp_args],
      name_prefix : '',
      name_suffix: 'so',
      install_dir: 'bin'
      ) 

However, bug still remains while running python script that used pysu2/pysu2ad module. There is a MPI related issue after calling Postprocessing in pysu2:

------------------------- Solver Postprocessing -------------------------
Deleted CNumerics container.
Deleted CIntegration container.
Deleted CSolver container.
Deleted CIteration container.
Deleted CInterface container.
Deleted CGeometry container.
Deleted CFreeFormDefBox class.
Deleted CSurfaceMovement class.
Deleted CVolumetricMovement class.
Deleted CConfig container.
Deleted nInst container.
Deleted COutput class.
-------------------------------------------------------------------------
[DN0a22c803:22646] *** Process received signal ***
[DN0a22c803:22646] Signal: Segmentation fault: 11 (11)
[DN0a22c803:22646] Signal code:  (0)
[DN0a22c803:22646] Failing at address: 0x0
[DN0a22c803:22646] *** End of error message ***
Segmentation fault: 11

The exact crash location is:

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000737e9bdce5a2
Exception Codes:       0x0000000000000001, 0x0000737e9bdce5a2
Exception Note:        EXC_CORPSE_NOTIFY

VM Region Info: 0x737e9bdce5a2 is not in any region.  Bytes after previous region: 3842065106339  Bytes before following region: 13412496251486
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      Stack                    70000ee7c000-70000eefe000 [  520K] rw-/rwx SM=PRV  thread 2
--->  GAP OF 0xfb164002000 BYTES
      MALLOC_TINY              7fb172f00000-7fb173000000 [ 1024K] rw-/rwx SM=PRV 

0   _pysu2.so                     	       0x116e2dfd0 CConfig::GetDiscrete_Adjoint() const + 12 (CConfig.hpp:8596)
1    _pysu2.so                     	       0x1173844ca CDriver::Postprocessing() + 8600 (CDriver.cpp:558)
2    _pysu2.so                     	       0x116bcaf0c _wrap_CDriver_Postprocessing + 199 (pySU2.cxx:10061)
0   libsystem_kernel.dylib        	    0x7ff801374d5a __select + 10
1   libopen-pal.40.dylib          	       0x10a42def6 select_dispatch + 182 (select.c:161)
2   libopen-pal.40.dylib          	       0x10a425719 opal_libevent2022_event_base_loop + 585 (event.c:1630)

Another observation is that when I was using mpi4py 3.0.3 and openmpi 4.1.3, I was able to rename _pysu2.dylib to _pysu2.so and import pysu2 module into python 3.9.1. However, when I use mpi4py 3.1.3, I wasn't able to import the renamed _pysu2.so into python3, the error message is very similar to the one reported with segmentation fault 11.

At this point, I think the potential approach is to enable python3 to import *.dylib module extension, since it seems that some *.dylib mpi shared library is also related to this. I tried to link .so extension for libopen-pal.40.dylib but that does not solve the issue. Any suggestion on this? Thank you!

My environment is : GNU -11.2.0_3 Python 3.9.1 Openmpi 4.1.3 SU2 7.3.0 feature_pysu2_fsi_adjoint mpi4py 3.0.3

[pcarruscag]

Thank you for the detective work @yiren6 :+1: We may need to consider changing the extension for OSX or create a symbolic link during the installation process. @patelha57 can you try the solution that yieren6 found?

As for the crash, I realize now that you are using the branch from PR #1300, it looks like that memory leak was fixed by @aa-g here https://github.com/su2code/SU2/commit/b2ddd9666b76b80ce5ce485b58dc5f97a27420b8 (at least that line 558 changed). When using code from a pull request you can have a look at the regression tests https://github.com/su2code/SU2/pull/1300 (bottom of the page) to get an idea if some issues might still be work in progress.

[patelha57]

Thank you for the detective work @yiren6 👍 We may need to consider changing the extension for OSX or create a symbolic link during the installation process. @patelha57 can you try the solution that yieren6 found?

As for the crash, I realize now that you are using the branch from PR #1300, it looks like that memory leak was fixed by @aa-g here b2ddd96 (at least that line 558 changed). When using code from a pull request you can have a look at the regression tests #1300 (bottom of the page) to get an idea if some issues might still be work in progress.

Hey Pedro! Yup, we worked side-by-side to figure this issue out, we're actually labmates (small world!). I can also confirm that pysu2 & pysu2ad are imported without errors on my OSX laptop. My environment is:

• Homebrew package manager
• GCC GNU compiler 11.2.0
• Python 3.9.12
• OpenMPI 4.1.3
• mpi4py 3.1.3
• swig 4.0.2

To summarize, the main change was to add name_suffix: 'so' option to /SU2_PY/pySU2/meson.build. This forces the output of _pysu2.so & _pysu2ad.so instead of OSX default of _pysu2.dylib & _pysu2ad.dylib for shared libraries.

[yiren6]

After some investigation I think the problem is related to how rpath was handled differently in OSX and Linux. There is a post on stackoverflow:

[https://stackoverflow.com/questions/19123623/python-runtime-library-dirs-doesnt-work-on-mac ](Python runtime_library_dirs doesn't work on Mac) Quote:

All of the libraries you're linking against must have install names that begin with "@rpath/". For example, if a library is called "libFoo.dylib", its install name should be "@rpath/libFoo.dylib". You can use "install_name_tool -id" to change the install name of a library.

There on I followed this link trying to change the rpath for _pysu2.dylib:

https://stackoverflow.com/questions/33991581/install-name-tool-to-update-a-executable-to-search-for-dylib-in-mac-os-x

However, I noticed that _pysu2.dylib does not have a field called "LC_RPATH". Some posts suggested to create a link image under system folder /usr/lib which is not possible for OSX even disabled Mac System Integrity Protection as the /usr/lib directory is read only.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is still a relevant issue please comment on it to restart the discussion. Thank you for your contributions.

[Avicene]

Bonjour à tous,

Je me permets de relancer cette issue (#1606) concernant l'impossibilité d'importer le module pysu2 sur Mac OSX.

Je vois que la dernière mise à jour concernait un problème lié au suffixe du fichier de bibliothèque partagée .dylib, et qu'en le changeant manuellement en _pysu2.so, le module Python pouvait être importé. Cependant, il semble qu'un autre problème soit apparu lié à MPI lors de l'exécution d'un script utilisant pysu2/pysu2ad​1​.

Est-ce que quelqu'un a trouvé une solution à ce problème ou a-t-il été résolu dans une version plus récente de SU2 ? Ce serait génial si quelqu'un pouvait partager des mises à jour ou des solutions de contournement.

Merci d'avance pour votre aide et votre temps.

[patelha57]

Hello @Avicene,

This is an open issue and has not been fixed in newer versions of SU2 as far as I am aware. By changing the dynamic library extension, we were able to import the pysu2 module but it was unusable. This was a while ago so I do not recall what the exact error or issue was, but all I know is that the module did not work (in serial or with MPI) after import on Mac OSX.

I can try to revisit this issue sometime this week to refresh my memory. But in any case, it would be good to join forces and try to find an adequate solution once and for all.

[bburke38]

Hello,

Has there been any further work on this bug? I just ran into it myself and was hoping a workaround may have been developed.

[patelha57]

No there has been no progress. I am focused on another deadline through August so I put it on the backburner. However I'd be happy to work with you to resolve this