calliope-project
Evaluating global expressions is broken when the `where` string does not filter out all missing values
What happened?
There seem to be issues currently with using NOT or OR/AND in the top level global_expressions where string related to the rework of _from_pyomo_expr() in pyomo_backend_model.py made in #551 . The expression below worked prior to #551 :
# This expression worked before the rework of _from_pyomo_expr()
# with a NOT in the top level expression where clause
flow_out_non_transmission_techs:
equations:
- expression: flow_out
foreach:
- nodes
- techs
- timesteps
- carriers
where: NOT base_tech=transmission
but now causes a TypeError: 'int' object is not callable in _from_pyomo_expr()
The issue seems to be specifically with NOT/OR/AND as doing a straight base_tech filter works. We can also work around it by moving the NOT logic to an equations where string but I wanted to flag the issue with the top level where filtering.
Other examples of expressions:
# A different approach to the expression that also fails with the OR
# in the top level where clause
flow_out_non_transmission_techs:
equations:
- expression: flow_out
foreach:
- nodes
- techs
- timesteps
- carriers
where: base_tech=supply OR base_tech=storage OR base_tech=conversion
# An expression that still works without any NOTs or ORs
# in the top level expression where clause
flow_in_demand_techs:
equations:
- expression: flow_in
foreach:
- nodes
- techs
- timesteps
- carriers
where: base_tech=demand
# This alternative option works with the NOT logic moved to the equations where strings
flow_out_non_transmission_techs_1:
where: flow_out
equations:
- where: NOT base_tech=transmission
expression: flow_out
foreach:
- nodes
- techs
- timesteps
- carriers
# Another alternative that works (just manually implementing an OR equivalent)
flow_out_non_transmission_techs_2:
equations:
- where: base_tech=supply
expression: flow_out
- where: base_tech=storage
expression: flow_out
- where: base_tech=conversion
expression: flow_out
foreach:
- nodes
- techs
- timesteps
- carriers
Which operating systems have you used?
- [ ] macOS
- [X] Windows
- [X] Linux
Version
v0.7.0.dev3
Relevant log output
No response
So, this is caused by flow_out not having a value for the demand technology group, so it is filled with its default value (the integer 0) in your main example. It isn't an issue of OR/NOT specifically, your other examples do/don't work because they aren't/are leaving those gaps. So, expression: flow_in works with base_tech=demand because all demand technologies have a value for flow_in. Your first non-working example (base_tech=supply OR base_tech=storage OR base_tech=conversion) works fine for me.
Moving NOT base_tech=transmission to the expression where string works because you have where: flow_out at the top level. You'll find that where: flow_out AND NOT base_tech=transmission will work fine at the top level.
Either there should have been an error raised in your main use-case, since the where string effectively left NaNs in the array (for demand techs), or it should have happily passed the unfiltered empty data as zeros along all the way to the end. I can't decide which would be the better result for the user...
