Failing to run parallel tests/examples on a parallel installation

[DeIonizedPlasma]

Hello,

I installed PyMFEM with parallel support using python setup.py install --mfem-branch=master --with-parallel --CC=cc --CXX=c++ --MPICC=mpicc --MPICXX=mpicx, and while the parallel examples in C seem to run properly, the PyMFEM ported examples all fail. When I execute python run_examples.py -parallel to compare PyMFEM and MFEM example outputs, every C example runs and properly generates output files, while every python example fails to generate any. Running any of the python parallel examples (ex0.py here) gives the same error:

Traceback (most recent call last):
  File "/home/joseph/Programs/MFEM-test/PyMFEM/examples/ex0p.py", line 24, in <module>
    import mfem.par as mfem
  File "/home/joseph/miniconda3/lib/python3.9/site-packages/mfem/par.py", line 12, in <module>
    from  mfem._par.globals import *
  File "/home/joseph/miniconda3/lib/python3.9/site-packages/mfem/_par/globals.py", line 13, in <module>
    from . import _globals
ImportError: /home/joseph/miniconda3/lib/python3.9/site-packages/mfem/external/par/lib/libmfem.so.4.4.0: undefined symbol: hypre_IntArrayDestroy

Running the example test script in serial mode works perfectly, so the error lies somewhere in the parallel installation and more specifically likely in hypre. I have tried doing a fresh re-installation of PyMFEM and have the same issue. I have also ensured that I properly set LD_LIBRARY_PATH according to my MPI installation, and tried re-installing without explicitly supplying the compilers to use.

I am running Ubuntu 22.04, Conda 4.13.0, Python 3.9.7, GCC 11.2.0, CMAKE 3.22.1.

[sshiraiwa]

Thank you for reporting the issue. It seems the one in the master seems has a issue with MFEM master branch. Working on a fix.

[sshiraiwa]

@DeIonizedPlasma when you have time, can you try the current master branch? The issue was that the wrapper code in master was generated with a slightly older version of MFEM. It became incompatible with MFEM master as MFEM master evolves.

[DeIonizedPlasma]

Apologies for the preliminary close. I attempted to run the same command but ran into a known GKlib issue (KarypisLab/GKlib#13) which required editing the gklib sha key to clone an older version. This time I am unable to get PyMFEM to successfully build at all, and the errors are not very helpful in illuminating why. I have attached the output and error messages, which I'm unable to discern the cause of failure from. log_out.txt log_err.txt

[sshiraiwa]

@DeIonizedPlasma Sorry for late response here. As discussed in other thread, we rolled back setup.py to point METIS 5.0.1. Please take a look at current master, and let us know if you find any issue. I don't have the same environment as you have setup right now, but here what I ran starting from a vanilla docker python image to build parallel.

sudo docker run -it python:3.9 bash
# git clone https://github.com/mfem/PyMFEM.git
# cd PyMFEM
# pip install SWIG cmake numba-scipy
# apt-get update
# apt-get install libopenmpi-dev libscalapack-openmpi-dev
# pip install mpi4py --install-option="CC=mpicc"
# python3 setup.py install --with-parallel --mfem-branch=master

or to use the package in PyPI, one can do the following instead of calling "python setup.py install -ta-li-la-li-la"

pip install mfem --install-option="--with-parallel" --install-option="--mfem-branch=master"

P.S. If something goes wrong, please run setup.py with --verbose option and upload it. Thank you.

[DeIonizedPlasma]

The above sequence of commands was enough to get it working on my machine. It works with docker and with miniconda's environment manager. I'm guessing my error was installing mpi4py without the mpicc argument, or that I was missing one of the mpi installs on the line above. Thank you!