Single Fracture Under Shear Compression from Advanced Examples does not work with MPI

[paveltomin]

[ptls@lxsphou0099 lagrangianContactMechanics]$ OMP_NUM_THREADS=1 mpirun -n 2 /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/build-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geosx -i ContactMechanics_SingleFracCompression_benchmark.xml
Num ranks: 2
Max threads: 1
GEOSX version: 0.2.0 (develop, sha1: 0eb9b90c2)
  - c++ compiler: gcc 10.4.0
  - openmp version: 201511
  - MPI version: Open MPI v4.1.5rc2, package: Open MPI root@hpc-kernel-03 Distribution, ident: 4.1.5rc2, repo rev: v4.1.5rc1-16-g5980bac633, Unreleased developer copy
  - HDF5 version: 1.12.1
  - Conduit version: 0.8.2
  - VTK version: 9.2.6
  - RAJA version: 2023.6.1
  - umpire version: 2023.6.0
  -  adiak version: ..
  - caliper version: 2.10.0
  - METIS version: 5.1.0
  - PARAMETIS version: 4.0.3
  - scotch version: 7.0.3
  - superlu_dist version: 6.3.0
  - suitesparse version: 5.7.9
  - Python3 version: 3.10.9
  - hypre release version: 2.29.0
Started at 2023-09-25 19:03:50.299150459
Adding Mesh: VTKMesh, CubeHex
Adding Geometric Object: Rectangle, fracture
Adding Geometric Object: Rectangle, core
Adding Geometric Object: Box, rightPoint
Adding Geometric Object: Box, leftPoint
Adding Geometric Object: Box, topPoint
Adding Geometric Object: Box, bottomPoint
Adding Geometric Object: Box, front
Adding Geometric Object: Box, rear
Adding Geometric Object: Box, xmin
Adding Geometric Object: Box, xmax
Adding Event: SoloEvent, preFracture
Adding Event: PeriodicEvent, solverApplications
Adding Event: PeriodicEvent, outputs
Adding Event: PeriodicEvent, timeHistoryCollection
Adding Event: PeriodicEvent, timeHistoryOutput
Adding Event: PeriodicEvent, timeHistoryCollection1
Adding Event: PeriodicEvent, timeHistoryOutput1
Adding Solver of type LagrangianContact, named lagrangiancontact
Adding Solver of type SolidMechanics_LagrangianFEM, named lagsolve
Adding Solver of type SurfaceGenerator, named SurfaceGen
   TableFunction: apertureTable
Adding Output: VTK, vtkOutput
Adding Output: TimeHistory, timeHistoryOutput
Adding Output: TimeHistory, timeHistoryOutput1
Adding Output: Restart, restartOutput
Adding Object CellElementRegion named Region from ObjectManager::Catalog.
Adding Object SurfaceElementRegion named Fracture from ObjectManager::Catalog.
VTKMesh 'CubeHex': reading mesh from /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/inputFiles/lagrangianContactMechanics/crackInPlane_benchmark.vtu
Generating global Ids from VTK mesh
Rank 1: Mesh sizes are: main = 14626
Rank 0: Mesh sizes are: main = 15350
VTKMesh 'CubeHex': generating GEOSX mesh data structure
Number of nodes: 40172
  Number of elems: 29976
             C3D8: 29976
Load balancing:   min   avg   max
(element/rank): 14626 14988 15350
No face block "faceElementSubRegion" was found in the mesh. Empty surface region was created.
regionQuadrature: meshBodyName, meshLevelName, regionName, subRegionName = CubeHex, Level0, Fracture, faceElementSubRegion
CubeHex/Level0/Fracture/faceElementSubRegion/fractureMaterial allocated 8 quadrature points
regionQuadrature: meshBodyName, meshLevelName, regionName, subRegionName = CubeHex, Level0, Region, 0_hexahedra
CubeHex/Level0/Region/0_hexahedra/rock allocated 8 quadrature points
regionQuadrature: meshBodyName, meshLevelName, regionName, subRegionName = CubeHex, Level0, Region, 1_hexahedra
CubeHex/Level0/Region/1_hexahedra/rock allocated 8 quadrature points
CubeHex: importing field data from mesh dataset
Time: 0.00e+00 s, dt: 1 s, Cycle: 0
***** ERROR
***** LOCATION: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/src/coreComponents/mesh/FaceElementSubRegion.cpp:415
***** Controlling expression (should be false): true
***** Rank 0: Mapping neighbor inconsistency detected for fracture faceElementSubRegion

** StackTrace of 9 frames **
Frame 0: geos::parallelTopologyChange::synchronizeTopologyChange(geos::MeshLevel*, std::vector<geos::NeighborCommunicator, std::allocator<geos::NeighborCommunicator> >&, geos::ModifiedObjectLists&, geos::ModifiedObjectLists&, int) 
Frame 1: geos::SurfaceGenerator::separationDriver(geos::DomainPartition&, geos::MeshLevel&, std::vector<geos::NeighborCommunicator, std::allocator<geos::NeighborCommunicator> >&, int, int, bool, double) 
Frame 2: geos::SurfaceGenerator::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 3: geos::SurfaceGenerator::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 4: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 5: geos::EventManager::run(geos::DomainPartition&) 
Frame 6: geos::GeosxState::run() 
Frame 7: main 
Frame 8: __libc_start_main 
Frame 9: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/build-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geosx 
=====

***** ERROR
***** LOCATION: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/src/coreComponents/mesh/FaceElementSubRegion.cpp:415
***** Controlling expression (should be false): true
***** Rank 1: Mapping neighbor inconsistency detected for fracture faceElementSubRegion

** StackTrace of 9 frames **
Frame 0: geos::parallelTopologyChange::synchronizeTopologyChange(geos::MeshLevel*, std::vector<geos::NeighborCommunicator, std::allocator<geos::NeighborCommunicator> >&, geos::ModifiedObjectLists&, geos::ModifiedObjectLists&, int) 
Frame 1: geos::SurfaceGenerator::separationDriver(geos::DomainPartition&, geos::MeshLevel&, std::vector<geos::NeighborCommunicator, std::allocator<geos::NeighborCommunicator> >&, int, int, bool, double) 
Frame 2: geos::SurfaceGenerator::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 3: geos::SurfaceGenerator::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 4: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 5: geos::EventManager::run(geos::DomainPartition&) 
Frame 6: geos::GeosxState::run() 
Frame 7: main 
Frame 8: __libc_start_main 
Frame 9: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/build-CPU-OPTO2-Hypre-GCC_10.2.0-ompi_hpcx-OMP-relwithdebinfo/bin/geosx 
=====

[jhuang2601]

This is the identified limitation of internal surface generator. If switching to external surface importer (#2580), this MPI issue would be fixed!

[paveltomin]

Same issue for VTK case https://github.com/GEOS-DEV/GEOS/blob/develop/inputFiles/lagrangianContactMechanics/ContactMechanics_UnstructuredCrack_benchmark.xml

Time: 0.00e+00 s, dt: 1 s, Cycle: 0

***** ERROR
***** LOCATION: /data/rpo_ptls/GEOSX/residual_flash/GEOS/GEOS/src/coreComponents/mesh/FaceElementSubRegion.cpp:414
***** Controlling expression (should be false): true
***** Rank 7: Mapping neighbor inconsistency detected for fracture FractureSubRegion

** StackTrace of 9 frames **
Frame 0: geos::parallelTopologyChange::synchronizeTopologyChange(geos::MeshLevel*, std::vector<geos::NeighborCommunicator, std::allocator<geos::NeighborCommunicator> >&, geos::ModifiedObjectLists&, geos::ModifiedObjectLists&, int) 

CC @TotoGaz

[paveltomin]

related to https://github.com/GEOS-DEV/GEOS/issues/841 ?