idyntree
idyntree copied to clipboard
robotology
Understand why iDynTree inverse kinematics crashes in Simulink and Matlab
I have encountered a problem in the iDynTree inverse kinematics (based on IPOPT) defined here.
I wrapped it in a blockfactory-based simulink block. When calling the solve method, it crashes with final output (note that IPOPT is at maximum verbosity)
******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
Ipopt is released as open source code under the Eclipse Public License (EPL).
For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************
This is Ipopt version 3.11, running with linear solver mumps.
NOTE: Other linear solvers might be more efficient (see Ipopt documentation).
Number of nonzeros in equality constraint Jacobian...: 126
Number of nonzeros in inequality constraint Jacobian.: 93
Number of nonzeros in Lagrangian Hessian.............: 0
Hessian approximation will be done in the space of all 31 x variables.
Running the debugger shows a segmentation fault at that line. Here is the stack trace before crashing:
1 mkl_blas_avx2_xdcopy 0x7f0bb1cce362
2 mkl_blas_dcopy 0x7f0bb0e79e45
3 __kmp_invoke_microtask 0x7f0dc33ed7d3
4 __kmp_invoke_task_func 0x7f0dc33b454a
5 __kmp_fork_call 0x7f0dc33b5d47
6 __kmpc_fork_call 0x7f0dc3375100
7 mkl_blas_dcopy 0x7f0bb0e79c39
8 mkl_blas.dcopy 0x7f0bb0c6960a
9 Ipopt::IpBlasDcopy(int, double const *, int, double *, int) 0x7f0b93917ef1
10 Ipopt::TNLPAdapter::GetStartingPoint(Ipopt::SmartPtr<Ipopt::Vector>, bool, Ipopt::SmartPtr<Ipopt::Vector>, bool, Ipopt::SmartPtr<Ipopt::Vector>, bool, Ipopt::SmartPtr<Ipopt::Vector>, bool, Ipopt::SmartPtr<Ipopt::Vector>, bool) 0x7f0b937d5ebe
11 Ipopt::GradientScaling::DetermineScalingParametersImpl(Ipopt::SmartPtr<Ipopt::VectorSpace const>, Ipopt::SmartPtr<Ipopt::VectorSpace const>, Ipopt::SmartPtr<Ipopt::VectorSpace const>, Ipopt::SmartPtr<Ipopt::MatrixSpace const>, Ipopt::SmartPtr<Ipopt::MatrixSpace const>, Ipopt::SmartPtr<Ipopt::SymMatrixSpace const>, Ipopt::Matrix const&, Ipopt::Vector const&, Ipopt::Matrix const&, Ipopt::Vector const&, double&, Ipopt::SmartPtr<Ipopt::Vector>&, Ipopt::SmartPtr<Ipopt::Vector>&, Ipopt::SmartPtr<Ipopt::Vector>&) 0x7f0b9383ab5a
12 Ipopt::StandardScalingBase::DetermineScaling(Ipopt::SmartPtr<Ipopt::VectorSpace const>, Ipopt::SmartPtr<Ipopt::VectorSpace const>, Ipopt::SmartPtr<Ipopt::VectorSpace const>, Ipopt::SmartPtr<Ipopt::MatrixSpace const>, Ipopt::SmartPtr<Ipopt::MatrixSpace const>, Ipopt::SmartPtr<Ipopt::SymMatrixSpace const>, Ipopt::SmartPtr<Ipopt::MatrixSpace const>&, Ipopt::SmartPtr<Ipopt::MatrixSpace const>&, Ipopt::SmartPtr<Ipopt::SymMatrixSpace const>&, Ipopt::Matrix const&, Ipopt::Vector const&, Ipopt::Matrix const&, Ipopt::Vector const&) 0x7f0b9388a429
13 Ipopt::OrigIpoptNLP::InitializeStructures(Ipopt::SmartPtr<Ipopt::Vector>&, bool, Ipopt::SmartPtr<Ipopt::Vector>&, bool, Ipopt::SmartPtr<Ipopt::Vector>&, bool, Ipopt::SmartPtr<Ipopt::Vector>&, bool, Ipopt::SmartPtr<Ipopt::Vector>&, bool, Ipopt::SmartPtr<Ipopt::Vector>&, Ipopt::SmartPtr<Ipopt::Vector>&) 0x7f0b9389329a
14 Ipopt::IpoptData::InitializeDataStructures(Ipopt::IpoptNLP&, bool, bool, bool, bool, bool) 0x7f0b93865f58
15 Ipopt::DefaultIterateInitializer::SetInitialIterates() 0x7f0b9382b720
16 Ipopt::IpoptAlgorithm::InitializeIterates() 0x7f0b9383dde6
17 Ipopt::IpoptAlgorithm::Optimize(bool) 0x7f0b93842503
18 Ipopt::IpoptApplication::call_optimize() 0x7f0b937c41a3
19 Ipopt::IpoptApplication::OptimizeNLP(Ipopt::SmartPtr<Ipopt::NLP> const&, Ipopt::SmartPtr<Ipopt::AlgorithmBuilder>&) 0x7f0b937c6c22
20 Ipopt::IpoptApplication::OptimizeNLP(Ipopt::SmartPtr<Ipopt::NLP> const&) 0x7f0b937c02f9
21 Ipopt::IpoptApplication::OptimizeTNLP(Ipopt::SmartPtr<Ipopt::TNLP> const&) 0x7f0b937c1260
22 internal::kinematics::InverseKinematicsData::solveProblem() 0x7f0b9421694c
23 WalkingControllers::WalkingIK::computeIK InverseKinematics.cpp 423 0x7f0b94e5c753
24 walkblocks::PrepareRobotRoutineBlock::prepareRobot prepareRobotRoutine.cpp 236 0x7f0b95157a6f
25 walkblocks::PrepareRobotRoutineBlock::output prepareRobotRoutine.cpp 310 0x7f0b95158368
26 mdlOutputs(SimStruct_tag *, int) 0x7f0b98682c87
27 ?? 0x7f0bdd093292
28 ?? 0x7f0bdd0988a5
29 ?? 0x7f0bdd06a703
30 ?? 0x7f0bda7c438d
31 ?? 0x7f0bdc821e53
32 ?? 0x7f0bdc8dde6a
33 ?? 0x7f0bdd15f375
34 ?? 0x7f0bdd1601b4
35 ?? 0x7f0be5ea1582
36 slexec::runtime::engine::ExecutionEngine::step() 0x7f0be5e85cd2
37 ?? 0x7f0bdd0f0094
38 ?? 0x7f0bdd0cd4d8
39 ?? 0x7f0bdd0cec0f
40 ?? 0x7f0bdd0cfd4c
41 ?? 0x7f0bdd0cff91
42 mwboost::shared_ptr<mwboost::unique_future<decltype ({parm#1}())>> cmddistributor::PackagedTaskIIP::invokeFunc<mwboost::function<void ()>>(mwboost::function<void ()> const&) 0x7f0de211796b
43 std::_Function_handler<mwboost::any (), std::function<mwboost::any ()> cmddistributor::PackagedTaskIIP::createFunc<mwboost::function<void ()>>(mwboost::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) 0x7f0de2117a58
44 mwboost::detail::function::function_obj_invoker0<std::function<mwboost::any ()>, mwboost::any>::invoke(mwboost::detail::function::function_buffer&) 0x7f0dcbaabdcc
45 iqm::PackagedTaskPlugin::execute(inWorkSpace_tag *) 0x7f0dcbaaba85
46 ?? 0x7f0dcb946b35
47 ?? 0x7f0dcba92f2d
48 ?? 0x7f0dcba778e8
49 ?? 0x7f0dd1671eaa
50 ?? 0x7f0dd1672c72
51 ?? 0x7f0dd167bc03
52 ?? 0x7f0dd167bd89
53 mnParser() 0x7f0dd167c288
54 ?? 0x7f0dcb9574f2
55 mwboost::shared_ptr<mwboost::unique_future<decltype ({parm#1}())>> cmddistributor::PackagedTaskIIP::invokeFunc<mwboost::function<void ()>>(mwboost::function<void ()> const&) 0x7f0de211796b
56 std::_Function_handler<mwboost::any (), std::function<mwboost::any ()> cmddistributor::PackagedTaskIIP::createFunc<mwboost::function<void ()>>(mwboost::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) 0x7f0de2117a58
57 mwboost::detail::function::function_obj_invoker0<std::function<mwboost::any ()>, mwboost::any>::invoke(mwboost::detail::function::function_buffer&) 0x7f0dcbaabdcc
58 iqm::PackagedTaskPlugin::execute(inWorkSpace_tag *) 0x7f0dcbaaba85
59 ?? 0x7f0dcb946b35
60 ?? 0x7f0dcba92f2d
61 ?? 0x7f0dcba75eba
62 ?? 0x7f0dcba76b2f
63 ?? 0x7f0dcb92de95
64 ?? 0x7f0dcb92e4b3
65 ?? 0x7f0dcb92ed24
66 ?? 0x7f0de0383bdd
67 start_thread pthread_create.c 463 0x7f0de13486db
68 clone clone.S 95 0x7f0de0ad988f
Note, this was happening on Ubuntu 18.04 with the system installed ipopt.
This issues seems related to https://projects.coin-or.org/Ipopt/ticket/285, that was migrated to https://github.com/coin-or/Ipopt/issues/285 . The issue was closed on the basis that the matlab interface is now maintained at https://github.com/ebertolazzi/mexIPOPT, but I now suspect that probably you would the same issue even using mexIPOPT.
The key observation is that the program is crashing while calling mkl_blas_avx2_xdcopy, that is a function of the MKL-specific BLAS implementation optimized for avx2 architectures. Clearly the system ipopt (when executed outside of matlab) does not use that function.
In theory different BLAS implementations should be ABI compatible. However, given that the system is crashing at the mkl_blas_avx2_xdcopy, what I suspect is happening is that the mkl BLAS used by MATLAB has some constraint on the alignment of the passed memory, and the allocator used in walking-controllers/IPOPT has no such constraint. If that is indeed the problem, we would need to force IPOPT to call the system blas that it usually use, instead of calling the internal mkl blas of Matlab.
I think we should try avoid to use the avx2 blas functions in Matlab with the following strategies:
- Set the
MKL_CBWRenv variable toSSE2(see https://software.intel.com/en-us/mkl-linux-developer-guide-setting-the-environment-variable-for-conditional-numerical-reproducibility and https://software.intel.com/en-us/onemkl-linux-developer-guide-specifying-code-branches#D66A7E19-D6C4-4A02-A8CE-DEE306C1CA3F) in the shell were you launch Matlab - If 1. does not work, try to set the env variables
BLAS_VERSIONto/usr/lib/x86_64-linux-gnu/liblapack.so.3andLAPACK_VERSIONto/usr/lib/x86_64-linux-gnu/liblapack.so.3in a shell and start Matlab in that shell
In both cases, even if the program continues to crash, it would be interesting to run in the matlab shell the commands version -blas and version -lapack.
Previous related comment that provides a bit more background:
By definition, all BLAS and LAPACK implementation share the same headers as they have the same ABI, so compiling against a different implementation does not change anything. In particular, the signature of the dcopy function (see http://www.netlib.org/lapack/explore-html/da/d6c/dcopy_8f.html) will always be the same. The problem is that I guess is that if
mkl_blas_avx2_xdcopyrequires 32 byte alignment (see https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/278730) and the allocator used by IPOPT/iDynTree does not guarantee this alignment, then the problem arise.However, I think that changing
BLAS_VERSION,LAPACK_VERSIONor evenMKL_CBWRis the a quite easy attempt, as it just involves setting an environmental variable.In the long run if the alignment is actually the problem, using a custom
mallocimplementation that ensures the required alignment.One thing that I realized yesterday night is that interesting this issue was not experienced by the people that recently used iDynTree MATLAB bindings for inverse kinematics (see https://github.com/robotology/idyntree/pull/633), perhaps they were just lucky with the alignment (if that is the problem). @lrapetti @claudia-lat