pyomo
pyomo copied to clipboard
Pyomo
Maximum recursion depth exceeded for (absurdly) long expressions
Summary
Some of the minlplib instances contain problems that have ridiculously long expressions that trigger a recursion error when initializing the model. It would be nice to solve these problems using pyomo.
Rationale
Description
Additional information
I have attached one such example (elec100.py) from the minlplib instance library, which has an objective function spanning over 5000 lines. A quick binary search on line numbers shows that if the expression extends beyond line 3437, the recursion error is thrown (see the file).
System information: Linux ubuntu 22.04 python 3.10.12 pyomo 6.7.1 elec100.txt
I have been unable to reproduce the recursion error with the attached elec100.txt (renamed to elec100.py) or from the original elec100.py from https://www.minlplib.org/download.html, and I have tried with Python 3.11 and 3.10, and versions of Pyomo back to 6.3.0. (just using pyomo solve --solver ipopt elec100.py)
Can you please update this issue with:
- the full exception trace you are seeing
- a minimal runnable example that reproduces the error you are seeing
Full exception trace that I see is one output line in the terminal: "RecursionError: maximum recursion depth exceeded during compilation"
Command I am using is "python elec100.py" I created a new environment on a new machine and observe the same behavior with elec100.py using the command "python elec100.py".
As minimal as I have found: attached is elec100.py without any of the constraints and one expression (the objective function) which I have truncated to the size that causes the maximum recursion depth error (this length is the same on both machines). Removing the last objective term
+ 1 / sqrt((m.x37 - m.x95)**2 + (m.x137
- m.x195)**2 + (m.x237 - m.x295)**2)
limits the length of the expression such that it does not cause the recursion error (in my observation).
This is an error being thrown by Python itself as it is trying to parse the exec100.py file. I am almost positive that it has nothing to do with Pyomo code (the error is happening before any Pyomo code is ever executed).
It might be because you are running a 32-bit interpreter?
Can you report the results from:
python -VV
and
python -c 'import sys; print(sys.maxsize)'
OK - I was able to reproduce this. This is unfortunately a Python issue and not a Pyomo issue. You can see this by commenting out the from pyomo.environ import * line in elec100.py and try running the file (you will see the same error from Python ... and this now is not executing a single line of Pyomo code).
The "fix" (i.e. hack workaround) is to up the Python interpreter recursion limit BEFORE trying to load / import the module:
import sys
sys.setrecursionlimit(10000)
import importlib
mod = importlib.import_module('elec100')
As there isn't anything we can really do in Pyomo to "fix" this, I am going to close this issue.
Another option is to use the "osil" format. There is an osil parser here: https://github.com/cog-imperial/suspect/blob/4a0e04d1be2f400416d508e632d81077a2b212ab/suspect/pyomo/osil_reader.py#L331.
Great suggestion @michaelbynum . For files with ABSURDLY long expressions (e.g., pedigree_sim2000.py with ~380000 line expression for the objective) the recursion limit of 10000 is not sufficient (and there may not even be enough room on the stack to prevent overflow). The osil reader works great!