Error when solving a block that has a constraint that contains a variable not in the block

[eslickj]

This is just a request for improvement in the exception message or logging when there is an exception that happens because you try to solve a block that has a constraint that references a variable that is not in the block. It looks like there is some code in https://github.com/Pyomo/pyomo/blob/27fc7d0049eb7bff781b3ecf7dd563dd5160d01b/pyomo/repn/plugins/ampl/ampl_.py#L1812 to provide a better message but what I see is below, and if you don't know why it happens it can be hard to understand.

Traceback (most recent call last):
  File "proto_gt.py", line 78, in <module>
    m, solver = main()
  File "proto_gt.py", line 69, in main
    m.fs.mx1.initialize()
  File "c:\users\eslickj\git\idaes-pse\idaes\generic_models\unit_models\mixer.py", line 940, in initialize
    res = opt.solve(blk, tee=slc.tee)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\base\solvers.py", line 575, in solve
    self._presolve(*args, **kwds)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\solver\shellcmd.py", line 198, in _presolve
    OptSolver._presolve(self, *args, **kwds)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\base\solvers.py", line 675, in _presolve
    **kwds)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\base\solvers.py", line 746, in _convert_problem
    **kwds)
  File "c:\users\eslickj\git\pyomo\pyomo\opt\base\convert.py", line 105, in convert_problem
    problem_files, symbol_map = converter.apply(*tmp, **tmpkw)
  File "c:\users\eslickj\git\pyomo\pyomo\solvers\plugins\converter\model.py", line 191, in apply
    io_options=io_options)
  File "c:\users\eslickj\git\pyomo\pyomo\core\base\block.py", line 1814, in write
    io_options)
  File "c:\users\eslickj\git\pyomo\pyomo\repn\plugins\ampl\ampl_.py", line 370, in __call__
    include_all_variable_bounds=include_all_variable_bounds)
  File "c:\users\eslickj\git\pyomo\pyomo\repn\plugins\ampl\ampl_.py", line 994, in _print_model_NL
    list(self_varID_map[id(var)] for var in linear_vars),
  File "c:\users\eslickj\git\pyomo\pyomo\repn\plugins\ampl\ampl_.py", line 994, in <genexpr>
    list(self_varID_map[id(var)] for var in linear_vars),
KeyError: 2041211129752

[blnicho]

This no longer throws an error so I'm closing it.

[jsiirola]

We have fixed this issue for the NL and LP interfaces. It is not clear if this is resolved in the direct / APPSI interfaces (I haven't tested it yet), and I know it is not yet working in the BARON / GAMS interfaces.

@blnicho, unless you object, I am going to reopen this at least until I finish the updated BAR / GMS writers.

[bknueven]

We have fixed this issue for the NL and LP interfaces. It is not clear if this is resolved in the direct / APPSI interfaces (I haven't tested it yet), and I know it is not yet working in the BARON / GAMS interfaces.

Variables seem to be okay in my testing, but parameters are not. Below is a simple non-working model:

from pyomo.environ import *

m = ConcreteModel()
m.p = Param(initialize=1, mutable=True)
m.b = Block(concrete=True)
m.b.x = Var()
m.b.c = Constraint(expr=(0, m.p*m.b.x, None))

s = SolverFactory("appsi_ipopt").solve(m.b)

Error message:

Traceback (most recent call last):
  File "/Users/bknueven/Software/pyomo_test.py", line 9, in <module>
    s = SolverFactory("appsi_ipopt").solve(m.b)
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/base.py", line 1568, in solve
    results: Results = super(LegacySolverInterface, self).solve(model)
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/solvers/ipopt.py", line 292, in solve
    self._writer.write(model, self._filename + '.nl', timer=timer)
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/writers/nl_writer.py", line 240, in write
    self.set_instance(model)
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/writers/nl_writer.py", line 76, in set_instance
    self.add_block(model)
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/base.py", line 1114, in add_block
    self.add_constraints(
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/base.py", line 1032, in add_constraints
    self._add_constraints(cons)
  File "/Users/bknueven/Software/pyomo/pyomo/contrib/appsi/writers/nl_writer.py", line 115, in _add_constraints
    cmodel.process_nl_constraints(
KeyError: 140619309905808

While the APPSI interfaces take care to collect up variables, named expressions, and external functions found in constraints, they do not collect parameters. See: https://github.com/Pyomo/pyomo/blob/d7d8287515e52603a5ef7e16767e48ab7681e214/pyomo/contrib/appsi/utils/collect_vars_and_named_exprs.py#L16 and https://github.com/Pyomo/pyomo/blob/d7d8287515e52603a5ef7e16767e48ab7681e214/pyomo/contrib/appsi/cmodel/src/expression.cpp#L1643

It would be straightforward to collect the parameters here in both places. But it looks like some non-trivial re-working of the APPSI base class would be necessary to take care that parameters are not added more than once, if they were collected out of constraint expressions.