AMICI, OpenMP and conda

[lcontento]

I get a SIGABRT which is apparently due to OpenMP.

__pthread_kill 0x00007fff6eac833a
pthread_kill 0x00007fff6eb84e60
abort 0x00007fff6ea4f808
__kmp_abort_process 0x0000000120d34440
__kmp_fatal 0x0000000120cf5d17
__kmp_register_library_startup() 0x0000000120cfd500
__kmp_do_serial_initialize() 0x0000000120cf6362
__kmp_get_global_thread_id_reg 0x0000000120cf6321
amici::AmiciApplication::runAmiciSimulations(amici::Solver const&, std::__1::vector<amici::ExpData*, std::__1::allocator<amici::ExpData*> > const&, amici::Model const&, bool, int) amici.cpp:211
amici::runAmiciSimulations(amici::Solver const&, std::__1::vector<amici::ExpData*, std::__1::allocator<amici::ExpData*> > const&, amici::Model const&, bool, int) amici.cpp:90
_wrap_runAmiciSimulations(_object*, _object*) amici_wrap.cxx:73854
cfunction_call_varargs 0x000000010f1e9920
_PyObject_MakeTpCall 0x000000010f1e8ad7
...

Possible explanation

I am on MacOS Catalina and use AMICI in a conda environment. The reason behind the crash seems to be an incompatibility between libomp (installed by brew) and libiomp5 (Intel's OpenMP library which comes with MKL which is bundled with numpy when you install it through conda). See https://community.intel.com/t5/Intel-C-Compiler/R-crashes-when-using-qopenmp-flag/td-p/1178305 for someone else having a similar problem. Indeed, if I replace -lomp with -liomp5 in https://github.com/AMICI-dev/AMICI/blob/6ef90cc515180fa8f904b88452ae39fecb79f6fa/python/amici/setuptools.py#L276-L280 everything works again.

How can we fix it?

One could check if the user is running in a conda environment and then check if he has installed MKL. (for reference: the following code checks if running in a conda env)

import sys, os
is_conda = os.path.exists(os.path.join(sys.prefix, 'conda-meta'))

Not sure if there are more elegant and general ways to do this with the python setup utilities (e.g., searching in all library paths for MKL). Also not sure if the same problem occurs also with Linux.

[dweindl]

@lcontento Thanks for reporting and investigating this. I think as a first step, I would just disable OpenMP by default when running in conda.

In general it would be great to handle conda environments better, i.e. looking for blas, hdf5, and openmp libraries there. I would probably go for a dedicated conda-forge recipe (#1052) that ensures all those dependencies are installed and sets the respective compiler/linker flags.

Unfortunately, looking for the right libraries during installation is quite a pain. Neither pkg-config nor cmake can be conveniently used here. At least I am not aware of any good / portable solutions.