Real-time bidding in double loop

[xiangao1]

Fixes

Summary/Motivation:

Solve real-time bidding problems and bid into the real-time market in the double loop.

We should merge #824 before this PR.

Changes proposed in this PR:

• Enable real-time bidding in the double loop
• When there is not enough dispatch/price signal from the market (Prescient), deactivate/unfix the corresponding tracking and bidding constraints/variables.
• Propose template (abstract) class for price forecaster.
• Update examples, docs, and tests.

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[codecov[bot]]

Codecov Report

Merging #855 (51ba188) into main (69972af) will increase coverage by 0.01%. The diff coverage is 92.12%.

@@            Coverage Diff             @@
##             main     #855      +/-   ##
==========================================
+ Coverage   70.18%   70.20%   +0.01%     
==========================================
  Files         573      574       +1     
  Lines       64217    64296      +79     
  Branches    12121    12127       +6     
==========================================
+ Hits        45071    45138      +67     
- Misses      16847    16864      +17     
+ Partials     2299     2294       -5     

Impacted Files	Coverage Δ
...pps/grid_integration/examples/thermal_generator.py	66.07% <50.00%> (+0.28%)	:arrow_up:
idaes/apps/grid_integration/forecaster.py	59.37% <59.37%> (-40.63%)	:arrow_down:
idaes/apps/grid_integration/examples/utils.py	86.66% <86.66%> (ø)
idaes/apps/grid_integration/bidder.py	94.78% <96.81%> (-0.68%)	:arrow_down:
idaes/apps/grid_integration/coordinator.py	97.44% <100.00%> (+1.11%)	:arrow_up:
idaes/apps/grid_integration/model_data.py	93.63% <100.00%> (-0.06%)	:arrow_down:
idaes/apps/grid_integration/tracker.py	100.00% <100.00%> (ø)
idaes/ver.py	61.53% <0.00%> (-4.62%)	:arrow_down:
idaes/core/dmf/util.py	55.55% <0.00%> (+1.19%)	:arrow_up:
idaes/core/dmf/codesearch.py	100.00% <0.00%> (+1.44%)	:arrow_up:
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[ksbeattie]

@xiangao1 I've put this on the IDAES August release board assuming it is NOT expected for the IDAES May release meaning it will not be in the DISPATCHE June release. Let me know if that is not correct!

[xiangao1]

@xiangao1 I've put this on the IDAES August release board assuming it is NOT expected for the IDAES May release meaning it will not be in the DISPATCHE June release. Let me know if that is not correct!

Thanks, @ksbeattie ! I anticipate this draft PR still needs some revisions. @adowling2 may have more to share about the plan.

[ksbeattie]

@xiangao1, will this be ready for the Aug release? As in, within the next week?

[xiangao1]

@ksbeattie I reviewed the conflicts listed here. Some of the conflicts are changes in the doc strings and import and some of them are introduced due to new capabilities. How should I address these conflicts?

[ksbeattie]

@ksbeattie I reviewed the conflicts listed here. Some of the conflicts are changes in the doc strings and import and some of them are introduced due to new capabilities. How should I address these conflicts?

The short answer is that they should be resolved so that both your changes and the changes from main are preserved.

But first a question: Are all the conflicts you are seeing here involving changes you've made in this branch?

I ask because this is a branch that is coming from a fork (yours) of a fork (Robby's) which can become rather confusing. It would be better for you to have your changes coming from a branch of your own direct fork from the primary idaes-pse upstream repo.

[xiangao1]

@ksbeattie Are all the conflicts you are seeing here involving changes you've made in this branch? Yes, all the conflicts are from this branch. On my local machine, my remotes are my fork and the idaes fork. I am not tracking Robby's fork anymore.

[xiangao1]

Hi @adowling2! Could you review this? I just formatted the code with Black.

[ksbeattie]

@xiangao1, @adowling2 We're planning on creating the IDAES Aug release candidate this week and this doesn't look ready.

[adowling2]

I should have a review done by the end of today.

[adowling2]

@radhakrishnatg @jghouse88 @jsiirola Any chance we could get a quick review?

[adowling2]

@ksbeattie we just need another reviewer

[radhakrishnatg]

Thank you @dguittet for reviewing the PR and for testing it with the nuclear case study! After this PR is merged, would you be able to open a PR in DISPATCHES with these changes (You will have to update your fork/branch to reflect the repo reorganization)?

@xiangao1 Looks like some tests are failing in the bidder code. Could you please take a look at it?

[lbianchi-lbl]

@radhakrishnatg @xiangao1 as far as I can understand the failing test is due to tx_utils (used for validate_and_clean_cost_curve() in bidder.Bidder._assemble_bids()) being imported conditionally only if egret is installed:

https://github.com/xiangao1/idaes-pse/blob/a52b074ab64f15aba073a657576ddd2856de75d0/idaes/apps/grid_integration/bidder.py#L1201-L1216

https://github.com/xiangao1/idaes-pse/blob/a52b074ab64f15aba073a657576ddd2856de75d0/idaes/apps/grid_integration/bidder.py#L22-L24

egret will be available in the developer installation (where most if not all of IDAES's optional dependencies are available), but not in the environment where the check "Run tests from site-packages (non-editable) installation" (which is designed for this exact purpose) is run.

Apart from skipping the tests that (directly or indirectly) result in Bidder._assemble_bids() being called with pytest.importorskip(), I would also consider adding a better error message (i.e. something more meaningful than the current NameError: name "tx_utils" is not defined) inside Bidder._assemble_bids() before tx_utils.validate_and_clean_cost_curve() is called, e.g.:

if not egret_available:
    raise RuntimeError("'egret' must be installed to use this functionality")

[xiangao1]

@radhakrishnatg @lbianchi-lbl optional dependency issue is addressed here: https://github.com/IDAES/idaes-pse/pull/855/commits/901574e0a1f41fa69a439a59597b7ddc8e43589a

Thanks for your help!