Updates path append in pformat_results

[samgdotson]

With this commit, pformat_results.py can find the data_processing module from any installation of Temoa. Previously, only sys.path.append('./data_processing') was used. This PR updates this line to sys.path.append(os.path.join(os.path.dirname(sys.path[0]), 'data_processing')) which adds data_processing to the absolute path.

[TemoaProject]

Thanks for submitting this pull request! By chance, are you running from a Windows or Linux machine? The pformat_results.py file resides in the temoa_model folder, so the "./" moves up a directory to the "temoa" directory, where the "data_processing" folder resides. However, we've only tested on Macs -- I'm wondering if the "./" notation doesn't work on other operating systems?

[samgdotson]

I've been running Temoa on a Linux machine and haven't tried it on Windows. Some of my colleagues have also encountered this issue and resolved it by explicitly adding temoa and data_processing to the python path (but this isn't a universal fix).

e.g. echo "export PYTHONPATH = "home/dotson/research/temoa/data_processing" >> ~/.bashrc

Of course, simple is better and I have also tried:

• ../data_processing
• data_processing

These also fail on linux machines.

If it turns out that none of these approaches work across all operating systems, it might be worthwhile to check the OS before importing. I.e.

if sys.platform == 'ubuntu':
    sys.path.append("path")
elif sys.platform == 'macintosh'
    sys.path.append("./data_processing")
...

[TemoaProject]

Sorry for the delay, I was trying to find ways to import the DB_to_Excel module without adding to the path, but it's a bit more complicated than I thought. This PR doesn't work on my machine because sys.path[0] actually points to where I have my environment installed, which is in a different location than the model itself. In order for the code to work, Temoa must be run from the top "temoa" directory. I've just made a commit that's similar to yours but uses os.getcwd() instead. Can you do a git pull and let me know if it works?

[samgdotson]

Custom imports have always given me a headache...

The line sys.path.append(os.path.join(os.getcwd(), 'data_processing')) points to the right location, but fails to import. I did a quick check with iPython from the temoa_model directory (I created the conda environment for Temoa and simply changed the name to temoa-new):

(temoa-new) dotson@ouroboros:~/research/temoa/temoa_model$ ipython
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys, os

In [2]: sys.path.append(os.path.join(os.getcwd(), 'data_processing'))

In [3]: from DB_to_Excel import make_excel
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-3-a5d6e9b37065> in <module>
----> 1 from DB_to_Excel import make_excel

ModuleNotFoundError: No module named 'DB_to_Excel'

I tried a couple of other things:

1. The current implementation

(temoa-new) dotson@ouroboros:~/research/temoa/temoa_model$ ipython
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys,os

In [2]: sys.path.append('./data_processing')

In [3]: from DB_to_Excel import make_excel
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-3-a5d6e9b37065> in <module>
----> 1 from DB_to_Excel import make_excel

ModuleNotFoundError: No module named 'DB_to_Excel'

and

2. A minor change (./ --> ../)

(temoa-new) dotson@ouroboros:~/research/temoa/temoa_model$ ipython
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys,os

In [2]: sys.path.append('../data_processing')

In [3]: from DB_to_Excel import make_excel

In [4]: exit()

However, in each of the above cases, running Temoa with

yes | python $TEMOA/temoa_model --config=data_files/run_bau.txt

yields

(temoa-new) dotson@ouroboros:~/research/pride/temoa-uiuc$ yes | python $TEMOA/temoa_model --config=data_files/run_bau.txt 
Traceback (most recent call last):
  File "/home/dotson/anaconda3/envs/temoa-new/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/dotson/anaconda3/envs/temoa-new/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dotson/research/temoa/temoa_model/__main__.py", line 28, in <module>
    from temoa_model import *
  File "/home/dotson/research/temoa/temoa_model/temoa_model.py", line 26, in <module>
    from temoa_run import *
  File "/home/dotson/research/temoa/temoa_model/temoa_run.py", line 50, in <module>
    from pformat_results import pformat_results
  File "/home/dotson/research/temoa/temoa_model/pformat_results.py", line 44, in <module>
    from DB_to_Excel import make_excel
ModuleNotFoundError: No module named 'DB_to_Excel'

Have you considered creating a setup.py file and putting Temoa on pypi? This way you could do

from temoa.data_processing.DB_to_Excel import make_excel
...

Otherwise, my only other idea is to give guidance in the README that users may need to change this import line or add temoa to their path explicitly.

Thanks for working on this!

[TemoaProject]

Thanks for your patience, and sorry this solution isn't working for you. I just test on our Linux cluster, and both the earlier version and my latest commit work fine. Can you try one more thing: can you execute directly from the temoa folder and run with your config file in the temoa_model folder? So the exact command would be:

$ python temoa_model/ --config=temoa_model/config_sample

where config_sample is running with temoa_utopia.sqlite, just as it comes from the repo?

Regarding pypi, we'd like to do that, but it might take some time.

BTW, I didn't know about the "yes" command, but that is awesome!

[samgdotson]

My apologies for the long delay! I tried running it from temoa and, unfortunately, the result is the same

(temoa-py3) dotson@ouroboros:~/research/temoa$ yes | python temoa_model/ --config=temoa_model/run_bau.txt 
Traceback (most recent call last):
  File "/home/dotson/anaconda3/envs/temoa-py3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/dotson/anaconda3/envs/temoa-py3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "temoa_model/__main__.py", line 28, in <module>
    from temoa_model import *
  File "temoa_model/temoa_model.py", line 26, in <module>
    from temoa_run import *
  File "temoa_model/temoa_run.py", line 50, in <module>
    from pformat_results import pformat_results
  File "temoa_model/pformat_results.py", line 44, in <module>
    from DB_to_Excel import make_excel
ModuleNotFoundError: No module named 'DB_to_Excel'

Thanks for taking the time to work on this!

[SutubraResearch]

Cleaning up the repo, and closing this PR as it's several years old.