iiasa
JRE error while running westeros_baseline in jupyter notebook
Problem description
Running westeros_baseline (without any modifications) in jupyter notebook (the Python [conda env:msg_env] kernel) caused error (the report in the log-file)
hs_err_pid43115.log
The error occurs when running cell #34: scenario.solve()
My best guess is that the message-ix installation is inconsistent with my iMac-env or with the installed GAMS. Unfortunately, I have no idea how to explore and fix the problem.
Versions
message-ix show-versions
ixmp: 3.3.0
message_ix: 3.3.0
message_ix_models: None
message_data: None
click: 8.0.1
dask: 2021.06.0
genno: installed
graphviz: None
jpype: 1.2.1
… JVM path: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre/lib/jli/libjli.dylib
openpyxl: 3.0.7
pandas: 1.2.4
pint: 0.17
xarray: 0.18.2
yaml: 5.4.1
iam_units: installed
jupyter: installed
matplotlib: 3.4.2
plotnine: None
pyam: 0.11.0
GAMS: 35.2.0
python: 3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:13:15)
[Clang 11.1.0 ]
python-bits: 64
OS: Darwin
OS-release: 20.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: (None, 'UTF-8')
I run the a copy ipynb Muhammad presented today and got another exception (copied below) which shows that my message-ix installation is faulty. I've checked GAMS (with their transport model): it appears to work OK. Please suggest how to trace the source of the problems (the above and below summarized). My best guess from the exception listing below is that jpype continues to be the trouble-maker.
scen.solve()
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in at.ac.iiasa.ixmp.objects.MsgScenario.toGDX()
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in at.ac.iiasa.ixmp.objects.Scenario.makeGamsDB()
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.<init>()
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.initializeAttributes()
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/_jpype.cpython-39-darwin.so in com.gams.api.GAMSWorkspace.verifySystemDirectory()
Exception: Java Exception
The above exception was the direct cause of the following exception:
com.gams.api.GAMSException Traceback (most recent call last)
<ipython-input-29-e0c31090e7e2> in <module>
----> 1 scen.solve()
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/message_ix/core.py in solve(self, model, solve_options, **kwargs)
563 :class:`.GAMSModel`.
564 """
--> 565 super().solve(model=model, solve_options=solve_options, **kwargs)
566
567 def add_macro(self, data, scenario=None, check_convergence=True, **kwargs):
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/core.py in solve(self, model, callback, cb_kwargs, **model_options)
1666 # Iterate until convergence
1667 while True:
-> 1668 model.run(self)
1669
1670 # Store an iteration number to help the callback
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/message_ix/models.py in run(self, scenario)
361
362 try:
--> 363 result = super().run(scenario)
364 finally:
365 # Remove the optfile regardless of whether the run completed
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/model/gams.py in run(self, scenario)
274 try:
275 # Write model data to file
--> 276 scenario.platform._backend.write_file(
277 self.in_file, ItemType.SET | ItemType.PAR, **s_arg
278 )
~/anaconda3/envs/msg_env/lib/python3.9/site-packages/ixmp/backend/jdbc.py in write_file(self, path, item_type, **kwargs)
576 # include_var_equ=False -> do not include variables/equations in
577 # GDX
--> 578 self.jindex[ts].toGDX(str(path.parent), path.name, False)
579 elif path.suffix == ".csv" and item_type is ItemType.TS:
580 models = set(filters.pop("model"))
com.gams.api.GAMSException: com.gams.api.GAMSException: GAMS system directory is both not specified and not found from your environment!
@marek-iiasa can you please check if you correctly specified GAMS in your environment variable as mentioned in installation instructions here.
yes, gams, as well as java, are in the path
(msg_env) Ba4:wrksp marek$ echo $PATH
/Users/marek/anaconda3/envs/msg_env/bin:/Library/Frameworks/GAMS.framework/Resources:/Applications/anaconda3/condabin:/anaconda3/bin:.:/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/Java/Home/bin:/Users/marek/bin:/Users/marek/src/mcma/ScriptsMac:/Library/TeX/texbin:/Library/Apple/usr/bin
I assume the including in the $PATH all one has to do, or ?
I did the install this morning on macOS: step1. I deleted Anaconda and GAMS. step2. rebooted machine. step3. installed GAMS 35.2 (including ticking the box to add to PATH) step4. installed anaconda3-2021.05 step5. installed messageix via anaconda following each step of the instructions with the addition of setting channel priority to strict. step5. downloaded tutorials step6. started jupyter notebook from the command-line ensuring environment 'message_env' is active.
Result -> no issues encountered as described above.
Below are the results from message-ix show-versions
ixmp: 3.3.0
message_ix: 3.3.0
message_ix_models: None
message_data: None
click: 8.0.1
dask: 2021.06.0
genno: installed
graphviz: None
jpype: 1.2.1
… JVM path: /Users/oliver/opt/anaconda3/envs/message_env/lib/jli/libjli.dylib
openpyxl: 3.0.7
pandas: 1.2.4
pint: 0.17
xarray: 0.18.2
yaml: 5.4.1
iam_units: installed
jupyter: installed
matplotlib: 3.4.2
plotnine: 0.8.0
pyam: 0.7.0
GAMS: 35.2.0
python: 3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:13:15)
[Clang 11.1.0 ]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: (None, 'UTF-8')
Many thanks, Oliver. Comparing your and my installations shall help me to narrow possible problems with my installation. The key "suspect" I noticed is JVM path: your points to the message_env while mine points to the JDK (which I installed and added to $PATH). Please comment on:
- Have you installed JDK and added its bin to your $PATH ?
- How your $JAVA_HOME is defined?
- Should JDK be added to $PATH or defining $JAVA_HOME is enough?
- Confirm that a tutorial (coffee_machine or westeros) opt-task was solved by gams; the show-versions in my installation didn't show errors; the exception occurred during scen.solve()
- The version of pyam differ (0.7 vs 0.11); I guess should not cause problems with running tutorials
- There is no plotnine in my env; this might be due to uses of pip (instead of conda) for my installation
- I noticed that we have slightly different OS-releases (19.6 vs 20.5) but I guess this unlikely causes problem with my installation.
I will reinstall message-ix when I'll be sure that our environments are as close as possible, i.e. after you will comment on the first four questions.
Marek,
As for questions 1, 2 and 3 -> No, I havent installed JDK. 4. Yes, the Westeros tutorial solved. 5. No, i believe this shouldnt create any issues. 6. Plotnine is only required for some reporting functionalities which are not called during the actual solving. 7. I have an older version of macOS, because the BigSur update caused macs of my type to be "bricked" and since then I have been hesitant to upgrade. But this also shouldnt make any issues.
Thanks again, Oliver,
Aha... Does it mean you have no $JAVA_HOME defined (as required by the installation instructions)?
Marek, no i didnt define $JAVA_HOME. This is only needed if Java is installed manually. I didnt do this so it is done so as part of the MESSAGEix installation process. So when calling which java, i get:
/Users/oliver/opt/anaconda3/envs/message_env/bin/java
Oliver, thanks again. I'll reinstall message_ix later today following your path, and report result here.