Unable to run binary RMG: ModuleNotFoundError

[mefuller]

Bug Description

Binary RMG installation is missing modules when called

How To Reproduce

Installing according to instructions at https://reactionmechanismgenerator.github.io/RMG-Py/users/rmg/installation/anacondaUser.html (but using miniconda since on storage-limited server) From rmg.py input.py, following error results immediately:

  File "/home/fuller/miniconda3/envs/rmg_bin/bin/rmg.py", line 48, in <module>
    from rmgpy.rmg.main import RMG, initialize_log, process_profile_stats, make_profile_graph
  File "/home/fuller/Code/RMG-Py/rmgpy/rmg/main.py", line 55, in <module>
    from rmgpy.rmg.model import Species, CoreEdgeReactionModel
  File "/home/fuller/Code/RMG-Py/rmgpy/rmg/model.py", line 41, in <module>
    import rmgpy.data.rmg
  File "/home/fuller/Code/RMG-Py/rmgpy/data/rmg.py", line 38, in <module>
    from rmgpy.data.base import ForbiddenStructures
  File "/home/fuller/Code/RMG-Py/rmgpy/data/base.py", line 44, in <module>
    from rmgpy.kinetics.uncertainties import RateUncertainty
  File "/home/fuller/Code/RMG-Py/rmgpy/kinetics/__init__.py", line 30, in <module>
    from rmgpy.kinetics.model import KineticsModel, PDepKineticsModel, TunnelingModel, \
ModuleNotFoundError: No module named 'rmgpy.kinetics.model'

Expected Behavior

RMG should run or have pulled in the necessary additional components in installation

Installation Information

Describe your installation method and system information.

• OS: EL7 (3.10.0-1127.19.1.el7.x86_64)
• Installation method: binary, miniconda
• RMG version information:

Name Version Build Channel

rmg 3.1.0 py37h1515d6f_0 rmg rmgdatabase 3.1.0 py37_0 rmg

[amarkpayne]

I think I have an idea of what is going on. It looks like you are calling the rmg binary as you wanted to do (see the top line in the traceback, the path is /home/fuller/miniconda3/envs/rmg_bin/bin/rmg.py, which is inside your conda environment for the binary). However, in the next lines, it is calling files from what I am guessing is your source installation (see files like /home/fuller/Code/RMG-Py/rmgpy/rmg/model.py).

Given this, I think this is caused by a PATH issue, where your source installation is higher on your path than the binary. It is possible though that this is simply that it can't find the binary files in your rmg_bin environment, and then falls back to the source installation which is lower on your PATH.

Can you look through your .bashrc file and comment out anything that would put the source installation on your PATH or PYTHONPATH? Then open up a new terminal and try running the RMG binary again (make sure you activate the conda environment).

[amarkpayne]

Just to follow-up, I also tried installing the binary version of RMG just now in a clean conda environment on Linux (Manjaro) and was able to run the minimal example file. I double checked all of my PATH variables to make sure that this was the only version on PATH, so I think the binary version is still okay.

Let me know if you are able to get anywhere with the PATH variable investigation on your end. Depending on what you find we can go from there.

[mefuller]

Good call - $PATH wasn't causing any issues, but it seems $PYTHONPATH was the problem: I have several locations added to it by default for development and the binary throws the above error with $PYTHONPATH set. Once I clear/unset it, the binary runs. However, after stopping and/or killing the RMG program, $PYTHONPATH is still unset, so it would be interesting to know what is happening here: where does the binary search by default and why does having (any/additional) locations in the $PYTHONPATH cause it to not find something? I'm trying to read the error trace and make sense of this still.

[amarkpayne]

So if I understand how python module searching works, whenever you try to import a module, python looks in the following places in this order:

1. The current directory where python was executed (if interactive) or the directory of the script being executed.
2. Any directory listed in PYTHONPATH, in order.
3. Locations adjacent to the python installation where the standard library and site packages are stored (e.g. anything in your conda environment, or anything you pip installed to that interpreter).

Given this, I think the following happened to you. First things first, you executed rmg.py input.py from the commandline. This was done in bash, which thinks that rmg.py must be some sort of executable. Therefore, it tries to find the executable with this name in every directory in order on PATH. When you activate your conda environment, conda places the bin folder of your environment to the very front of PATH. This folder is where we place the rmg.py executable in the binary installation, so bash finds this executable first.

It turns out though that the first line of this executable is a directive to run the file in python, so python gets called here. Once running this script in python, the interpreter will hit various import statements. Now, the python interpreter must find these modules by searching for them in the order described above. The files you want are actually stored as site packages in the binary installation, found relative to the python interpreter (which given your PATH from conda is the python interpreter in your conda environment). However, this location is not searched before anything on PYTHONPATH. Therefore, if you have an rmgpy installation on PYTHONPATH, it will use this one instead.

Once it has imported rmgpy once, it will only search for rmgpy.* packages here (it will not keep searching PYTHONPATH or site packages if it can't find a file). Therefore, even though you have the files you need in site packages, it cannot find them now (this is actually a good thing, as it keeps you from using files from perhaps incompatible versions, which would be a nightmare to debug).

As to why you could find some files from your source installation but not others, my best guess is that you had not run make clean; make in a while, as rmgpy.kinetics.model is a cythonized file. If it had not been compiled, the interpreter cannot find this file.

That was probably a bit longer than you were looking for, but I think that is the complete description as to what happened.

As for how you might proceed, you might want to look into pip -e ./ functionality. This allows you to install python packages in development mode. This is better than storing things on PYTHONPATH because this installation is only added to a specific python interpreter.

For example, to add you RMG-Py folder for your source installation so that it only shows up on your source installation's conda environment, do the following.

1. Activate the conda environment that you only use for your source installation of RMG-Py (e.g. conda activate rmg_env)

2. cd to your RMG-Py folder.

3. Run make clean; make to make sure everything is compiled first

4. Run pip install -e ./. This adds any python packages in the current directory into your site packages in development mode (by linking files to site packages rather than directly copying them). This way, you can edit these files in your source directory without having to continually copy them over to site packages.

[github-actions[bot]]

This issue is being automatically marked as stale because it has not received any interaction in the last 90 days. Please leave a comment if this is still a relevant issue, otherwise it will automatically be closed in 30 days.