Support for p-refinement in PyMFEM

[gillette7]

I'm working with some folks on a project where we'd like to use p-refinement within PyMFEM. p-refinement is supported in the upcoming MFEM release (v. 4.3) but we'd like to develop the PyMFEM hooks now. My student Justin Crum who is at LLNL (virtually) for the summer will start tackling this. Any concerns or questions, please contact me. Thanks!

[sshiraiwa]

Thank you for letting us know the plan. I think we should start a PyMFEM branch tracking the mfem branch for 4.3. I suppose this is "nest" branch. Is it correct? -> @tzanio

[tzanio]

It may be better to track mfem-4.3-dev, which I update periodically (next has changes that are still being tested and may not work).

[sshiraiwa]

FYI. mfem-4.3-dev branch was made and a few changes are made including...

• setup.py has a new option --mfem-branch to specify which mfem branch to be downloaded.
• couple of adjustment in SWIG interface files are make to run ex1.py and ex1p.py
• no particular efforts are made yet to support newly added *.hpp in mfem-4.3 ;D

Note:

1. I can not find the actual implementation of

    Mesh(Mesh *orig_mesh, const Array<int> &ref_factors, int ref_type),

which is declared in mesh.hpp. At moment, mesh.i is adjusted to ignore this symbol.

2. Parallel version complains that

   swig/python detected a memory leak of type 'HYPRE_BigInt *', no destructor found.

which needs to be fixed.

[tzanio]

Thanks @sshiraiwa, let me take a look at the constructor.

Do you know where the memory leak is from?

[sshiraiwa]

Memory leak is an error on the wrapper side. HYPRE_BigInt was not properly recognized as a typedef. It seems fixed now.

[tzanio]

Ok, @pazner just fixed the first issue in https://github.com/mfem/mfem/pull/2189/commits/72fcd82618e0e2565189e15d5e2980ea86a78c9b (that constructor was removed)

[v-dobrev]

Maybe we can now mark as done the reminder I left here: https://github.com/mfem/mfem/pull/2189#issue-619350974

[Justincrum]

I'm getting a swig error when I try to do python3 setup.py install --mfem-branch mfem-4.3-dev. Is that something on my end that is messed up?

[sshiraiwa]

I'm getting a swig error when I try to do python3 setup.py install --mfem-branch mfem-4.3-dev. Is that something on my end that is messed up?

Did you clone PyMFEM and switch to mfem-4.3-dev branch? Can you paste the full error log? Thank you.

[Justincrum]

Yes, I clone PyMFEM and then swap to the mfem-4.3-dev branch.

So if I enter python3 setup.py install --mfem-branch mfem-4.3-dev,

then the error I get is

Traceback (most recent call last):
  File "setup.py", line 97, in <module>
    swig_command = find_command('swig') if os.getenv("SWIG") is None else os.getenv("SWIG")
NameError: name 'find_command' is not defined

[Justincrum]

I think I got it to work. The find_command function was defined in my file after it gets called, so I needed to move that up to just before the first time it is called, and then the value for the "mfem" key that the setup file is trying to run git clone from wasn't working so I had to copy and paste in a new clone string from the mfem github repo.

[sshiraiwa]

Understood. swig_command is used only inside generate_wrapper. It should have been defined in the function instead of in the global scope. I just pushed a commit to fix this. Let me know if you encounter any other issues.

[Justincrum]

@sshiraiwa I think I've got either a misunderstanding in how swig works or found another issue.

When I run python3 setup.py install --mfem-branch drl4marking, there's a function that is missing from the python side (drl4marking is a branch that a few of us are working on for reinforcement learning).

Specifically, we have a function in the gridfunction.cpp/hpp files called ProlongToMaxOrder. I thought that when I run the setup.py file with the correct branch of mfem, that swig would automatically generate the wrapper for this function, but it is not. Do you know if this is an error on my end?

[sshiraiwa]

—branch simply download the specified mfem branch and build it.

If you have only one change like this, you may be able to do

touch mfem/*/gridfunc.i
python setup.py install --run-swig —with-parallel

to update a wrapper code. Otherwise, you could regenerate everything

python setup.py clean  —swig
python setup.py install --run-swig —with-parallel
python setup.py install --with-parallel

[gillette7]

I'm getting a strange error when running python setup.py install:

It finishes compiling mfem but then this happens:

Moving to a directory : /Users/gillette7/Desktop/projects/drl_SI/PyMFEM
Moving to a directory : /Users/gillette7/Desktop/projects/drl_SI/PyMFEM/mfem/_ser
calling ... /Users/gillette7/.pyenv/versions/drl4mkg/bin/python setup.py build_ext --inplace
building serial version
numpy inc /Users/gillette7/.pyenv/versions/drl4mkg/lib/python3.7/site-packages/numpy/core/include
running build_ext
building '_io_stream' extension
cc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -DTARGET_PY3=1 -I/Users/gillette7/.pyenv/versions/drl4mkg/mfem/ser/include -I/Users/gillette7/.pyenv/versions/drl4mkg/mfem/ser/include/mfem -I/Users/gillette7/.pyenv/versions/drl4mkg/lib/python3.7/site-packages/numpy/core/include -I/Users/gillette7/.pyenv/versions/drl4mkg/include -I/Users/gillette7/.pyenv/versions/3.7.3/include/python3.7m -c io_stream_wrap.cxx -o build/temp.macosx-10.14-x86_64-3.7/io_stream_wrap.o -std=c++11 -DSWIG_TYPE_TABLE=PyMFEM
clang: error: no such file or directory: 'io_stream_wrap.cxx'
clang: error: no input files
error: command 'cc' failed with exit status 1
Failed when calling command: /Users/gillette7/.pyenv/versions/drl4mkg/bin/python setup.py build_ext --inplace
Traceback (most recent call last):
  File "setup.py", line 1139, in <module>
  File "setup.py", line 1135, in main
  File "setup.py", line 1131, in run_setup
  File "/Users/gillette7/.pyenv/versions/drl4mkg/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 945, in run
  File "/Users/gillette7/.pyenv/versions/drl4mkg/lib/python3.7/site-packages/setuptools/command/install.py", line 65, in run
    orig.install.run(self)
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "setup.py", line 987, in run
  File "setup.py", line 668, in make_mfem_wrapper
  File "setup.py", line 210, in make_call
  File "/Users/gillette7/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/Users/gillette7/.pyenv/versions/drl4mkg/bin/python', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1.

Any ideas how to fix this? Thanks!

[sshiraiwa]

Did you clean the wrapper code? If so, did you generate them?

When you have already build MFEM, then usual steps to re-generate wrapper and build them would be $ python setup.py clean --swig <---- this delete all wrapper code $ python setup.py install --swig (--with-parallel if you want) <---- this re-generates all *_wrap.cxx files $ python setup.py install --skip-ext (--with-parallel if you want)

As a side note, the current test branch is close to what is going to PyMFEM4.3, although I still needs to generate a couple of Python examples for new examples.

[gillette7]

Thanks @sshiraiwa - that worked. I didn't know about --skip-ext, which seemed to be the key step I was missing.