"Fatal Python error: PyThreadState_Get: no current thread" due to wrong Python library loaded by CMAKE

[nunoguedelha]

Conditions for reproducing the issue:

The issue occurs on MacOSX when using a brewed Python version, for instance 2.7.11

1. building iDynTree with IDYNTREE_COMPILE_TESTS ON, IDYNTREE_USES_PYTHON_VERSION set to 2.7.11
2. In the build directory launch the Python tests:

ctest -VV -C Debug -R PythonGeometryTests

We get the error:

test 52
    Start 52: PythonGeometryTests

52: Test command: /usr/local/bin/python "geometry.py"
52: Test timeout computed to be: 1500
52: Fatal Python error: PyThreadState_Get: no current thread
1/1 Test #52: PythonGeometryTests ..............***Exception: Other  2.19 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   2.22 sec

The following tests FAILED:
     52 - PythonGeometryTests (OTHER_FAULT)
Errors while running CTest

[nunoguedelha]

Find below the Apple crash report :

Process: Python [76904] Path: /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 2.7.11 (2.7.11) Code Type: X86-64 (Native) Parent Process: ctest [76903] Responsible: Terminal [223] ... OS Version: Mac OS X 10.10.5 (14F1808) ... Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Application Specific Information: abort() called . Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libsystem_kernel.dylib 0x00007fff89f01286 __pthread_kill + 10 1 libsystem_c.dylib 0x00007fff81b599ab abort + 129 2 org.python.python 0x000000010b1a62a3 Py_FatalError + 49 3 org.python.python 0x000000010b1a5785 PyThreadState_Get + 28 4 org.python.python 0x000000010b1a2522 Py_InitModule4_64 + 62 5 _iDynTree.so 0x000000010aa396d9 init_iDynTree + 73 (iDynTreePYTHON_wrap.cxx:75559) 6 org.python.python 0x000000010a4fb98b _PyImport_LoadDynamicModule + 140

[nunoguedelha]

Call tree before crash:

➡️ geometry.py : import iDynTree ➡️ iDynTree.py : import _iDynTree ➡️ iDynTreePYTHON_wrap.cxx : init_iDynTree( ) ➡️ modsupport.h : Py_InitModule4_64( )

file modsupport.h is located in the include path of the system Python, also including the file patchlevel.h : .../System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/

The system Python version defined in patchlevel.h happens to be 2.7.10. This will be the version compared to the Python version set when compiling iDynTree, and running the Python script, which explains the crash. The error message is explained here, and would occur when we

... use a version of Python that is not binary compatible with the one used to compile the Python bindings.

Whatever Python version is set in IDYNTREE_USES_PYTHON_VERSION, the same system include folder is used. Actually, for any brewed Python version used, the cmake find_package command (find_package(PythonLibs <version> [EXACT] [QUIET] [MODULE])) is failing to find the Python library. Calling find_package giving the explicit path to the library libpython2.7.dylib wouldn't work either. This seems to be an old and recurrent issue as per the long thread here https://github.com/Homebrew/legacy-homebrew/issues/25118 (without a clear solution).

In my case, clearing the CMakeCache.txt, setting IDYNTREE_USES_PYTHON_VERSION = 2.7.10 and rebuilding solved the issue. But it's a workaround and probably deserves further analysis.

@traversaro , any ideas?

[traversaro]

In a nutshell the problem is that the python interpreter returned by find_package(PythonInterp ...) is different from the library found by find_package(PythonLibs ...), am I right?

[traversaro]

Which version of CMake are you using? Since CMake 3.5 we should be ensured that the two calls return consistent python versions, see https://cmake.org/cmake/help/v3.5/module/FindPythonLibs.html?highlight=findpython#module:FindPythonLibs .

[nunoguedelha]

@traversaro , about find_package... yes, the PythonLibs found will not match the PythonInterp because cmake never finds the libraries related to a python interpreter installed through brew. If the found python interpreter is the OSX system one, then cmake will find the respective library. About the cmake version. I was using version 3.4.1, just updated to 3.6.1. Anyway, I've cleaned-up my Python packages, removing Anaconda, brew, and just keeping one Python system version (2.7.10).

[traversaro]

Related problem : https://github.com/robotology/idyntree/issues/153 .

[traversaro]

This error is now reproducible: https://github.com/robotology/idyntree/pull/568/checks?check_run_id=206937776 / related commits: https://github.com/robotology/idyntree/pull/568/commits/e03f88a7a85fb1982688b43e6cfd6dfe19db41e9 .