MGR: ERROR: Controlling expression (should be false): temp != 0 (in HypreMatrix.cpp)

[paveltomin]

Case: coupled_benchmark (2).zip

Running with 2400 MPIs instead of 1200 MPIs leads to the following error:

Time: 0.00e+00 s, dt: 10000 s, Cycle: 0

    Attempt:  0, ConfigurationIter:  0, NewtonIter:  0
    ( Rflow ) = ( 2.95e+00 ) ;     ( Rsolid ) = ( 1.07e+02 ) ;     ( R ) = ( 1.07e+02 ) ; 
{ 3, 3 }
***** ERROR
***** LOCATION: /dev/shm/mtml/src/GEOS/GEOS/src/coreComponents/linearAlgebra/interfaces/hypre/HypreMatrix.cpp:846
***** Controlling expression (should be false): temp != 0
***** Rank 1474: 
Expected temp == 0
  temp = 1
  0 = 0


** StackTrace of 14 frames **
Frame 0: geos::HyprePreconditioner::setupPreconditioningMatrix(geos::HypreMatrix const&) 
Frame 1: geos::HyprePreconditioner::setup(geos::HypreMatrix const&) 
Frame 2: geos::HypreSolver::setup(geos::HypreMatrix const&) 
Frame 3: geos::SolverBase::solveLinearSystem(geos::DofManager const&, geos::HypreMatrix&, geos::HypreVector&, geos::HypreVector&) 
Frame 4: geos::SolverBase::solveNonlinearSystem(double const&, double const&, int, geos::DomainPartition&) 
Frame 5: geos::SolverBase::nonlinearImplicitStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 6: geos::SolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 7: geos::CoupledSolver<geos::CompositionalMultiphaseBase, geos::SolidMechanicsLagrangianFEM>::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 8: geos::SolverBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 9: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 10: geos::EventManager::run(geos::DomainPartition&) 
Frame 11: geos::GeosxState::run() 
Frame 12: main 
Frame 13: __libc_start_main 
Frame 14: /chap/geos/0.2.0/x86_64/RHEL7/install-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geos 

Comments from @klevzoff: https://github.com/GEOS-DEV/GEOS/blob/2027eae63fe5424cf342b4ae609862b9942f1aea/src/coreComponents/linearAlgebra/interfaces/hypre/HypreMatrix.cpp#L844-L846 That condition should be correct for a purely mechanics stiffness matrix, but is a bit too restrictive otherwise We shouldn't be invoking separate component filter on the coupled matrix?