Add modelinstance support

[Wegatriespython]

This is an experimental branch which adds support for the Gamsmodelinstance API which provides a persistent model object for use.

Right now this is quite experimental, hence the code is not up to quality. The API for allowing user access is also not final. Putting it here for reference and feedback. For most models, model generation times dominate solving times. This means when a sequence of scenarios are to be evaluated in an ensemble a large fraction of the time is spent in the generation of the model matrix. Modelinstance provides a persistent object with an in-memory database for modifiable parameters via sync_db.

In theory this allows for gains = 1 + time_build/time_solving. For the baseline model time_build is ~500s and solve is 200s which yields a gain of 3.5X for large number of runs. Additionally the model instance passes the previous solution's basis to the next solution. In theory then a barrier followed by Simplex methods with advanced basis can approach even higher performance gains. But this requires dynamic solver parametrization maybe ML methods could help here.

How to review

At this point feedback and usage would be appreciated

PR checklist

Experimental so this will not apply till at least an early Alpha stage is reached.

~~- [ ] Continuous integration checks all ✅~~

~~- [ ] Add or expand tests; coverage checks both ✅~~ ~~- [ ] Add, expand, or update documentation.~~ ~~- [ ] Update release notes.~~

[codecov[bot]]

Codecov Report

:x: Patch coverage is 2.63158% with 111 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 96.9%. Comparing base (413ef71) to head (b5ae1f3). :warning: Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
ixmp/model/gams.py	1.8%	108 Missing :warning:
ixmp/core/scenario.py	25.0%	3 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #605     +/-   ##
=======================================
- Coverage   98.5%   96.9%   -1.7%     
=======================================
  Files         51      51             
  Lines       6300    6413    +113     
=======================================
+ Hits        6211    6218      +7     
- Misses        89     195    +106     

Files with missing lines	Coverage Δ
ixmp/core/scenario.py	98.0% <25.0%> (-1.2%)	:arrow_down:
ixmp/model/gams.py	58.6% <1.8%> (-38.8%)	:arrow_down:

... and 3 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[glatterf42]

Thanks for opening this PR! In principle, this sounds like a very useful addition :) I'd like to make sure that these changes reach main as soon as possible to avoid a situation where new users are directed to this branch, which might fall out of sync with main (as it already has), causing incompatibilities, etc.

To me, it looks like the changes here need to be better integrated within our stack before all users can profit from them. For example, we should not set MESSAGE variable dimensions here as this constitutes a stack-level violation: ixmp should be entirely usable without any knowledge of message_ix, so everything that is only relevant for message_ix should live in that package instead of here.

Do you know when you will have time to continue work on this?

[Wegatriespython]

Hey Fridolin thanks for the feedback, as it stands its not ready for use yet so I have not pointed it to users. One roadblock before implementation is that I had some inconsistent behaviour when testing it with Westeros models vs full size models. Hopefully early next year I can validate that the behaviour on production models is consistent. After which I can work on integrating it as you recommend.