Migrating away from GAMS

[glatterf42]

What is this about?

Since GAMS is proprietary software, receiving one of the expensive licenses is often a bottleneck for using message_ix, especially for researchers from the Global South. It would thus greatly benefit our inclusivity and bolster our open source spirit if we moved away from GAMS.

There are several replacement options: pyomo and linopy come to mind most readily. I saw a presentation about linopy and a performance comparison during the OpenMod workshop hosted at IIASA last year, which is why I chose linopy (using the HiGHS solver underneath) to implement a proof of concept for the ixmp4 transport tutorial. A similar migration could be done for the actual MESSAGE GAMS code, though this comprises a lot of code and would thus likely require a significant time investment.

During today's weekly meeting, I brought up my wish to eventually perform this migration. Several colleagues shared their experiences and made suggestions right away:

• @behnam-zakeri shared that we could look into gamspy, too. This, together with GAMS' move to make licenses free for academic institutions would present an alternative to using linopy/HiGHS. We would still have a python interface to the GAMS code, while possibly avoiding the actual rewriting of our GAMS code and likely avoiding the workflow problem presented by @SiddharthJoshi-Git below. The downside would be that our results would still not be completely publicly reproducible and we would still support the closed-source GAMS.
• @SiddharthJoshi-Git shared part of his usual workflow: for quick assessment of results or copying of data, he often uses the .gdx files, which seem to be much more performant than dumping the results to Excel or .csv. Our migration should take this into account and provide ways to quickly and conveniently access the same information stored in the .gdx as well as the .log and .lst files.
• @yiyi1991 expressed her support for this migration and stated that the OseMOSYS team has already done something similar. This might be a place to look for examples as well as an opportunity for collaboration.
• @khaeru noted that we would need to ensure before migration that all GAMS features we use are also available with our choice of tools and that performance is at least at comparable levels.

All of this means that we're not likely to start this migration anytime soon, much less finish it, but I wanted to record this discussion. For future reference and for you, dear enthusiastic community member: please reach out to us to get information on the current status of this project! Any help will be greatly appreciated :heart:

[awais307]

I would like to add one experience where we used GAMS Base Module (academic license) at UVic and then use open-source solvers such as CBC. I briefly checked the results and the results are looking similar on a country scale model with 13 nodes.

[glatterf42]

I would like to add one experience where we used GAMS Base Module (academic license) at UVic and then use open-source solvers such as CBC. I briefly checked the results and the results are looking similar on a country scale model with 13 nodes.

Thanks @awais307! A clarifying question from my side, if I may: Do you use GAMS and CBC simultaneously, i.e. for the same Scenario.solve() workflow? Or do you use either GAMS or CBC for each individual Scenario.solve() workflow and you compared both versions for one model now? If the former, what do you consider the "GAMS Base Module" and which functions does it fulfill (contrasting to CBC)? Also, could you point to a version of the code that does this somewhere on GitHub?

[criscfer]

Hello @glatterf42 I work closely with @awais307 as a software developer. This issue has recently resurfaced into our team's discussions as we navigate some restrictions that come with using GAMS as an optimization engine for MESSAGE-ix, and that made us wonder if any significant progress has been done on this migration effort.

Is IIASA still considering GAMSpy, Pyomo, or any other alternative to the classic GAMS engine for modernizing the MESSAGE-ix framework? Can you elaborate on the necessary steps that this migration would need to go through? From my understanding, migrating away from pure GAMS into GAMSpy, for example, would mean re-writing the core MESSAGE-ix constraints and objective function from .gms files to .py files. Would all the GAMS files found in the message_ix folder need to be re-written?

[criscfer]

Thanks @awais307! A clarifying question from my side, if I may: Do you use GAMS and CBC simultaneously, i.e. for the same Scenario.solve() workflow? Or do you use either GAMS or CBC for each individual Scenario.solve() workflow and you compared both versions for one model now? If the former, what do you consider the "GAMS Base Module" and which functions does it fulfill (contrasting to CBC)? Also, could you point to a version of the code that does this somewhere on GitHub?

Hi @glatterf42, sorry it took us so long to respond to you here. We use CBC as the solver for MESSAGE-ix through the auxiliary_settings.gms file in GAMS. I believe that all the steps in the workflow are handled by and through GAMS, except for the optimization, where CBC is called to optimize the problem and return the solution to GAMS. I'm not aware of any examples of this kind of solver implementation available in GitHub

[khaeru]

@criscfer some responses which I hope are helpful:

Can you elaborate on the necessary steps that this migration would need to go through? From my understanding, migrating away from pure GAMS into GAMSpy, for example, would mean re-writing the core MESSAGE-ix constraints and objective function from .gms files to .py files.

Roughly, yes, this is a correct assessment. A few details in case they are not clear from the docs:

• Most of the model/solver interface code is in the ixmp package/repo. There, a class named GAMSModel does the following:
  • Write the contents of a single ixmp.Scenario (or a subclass like message_ix.Scenario) to a GAMS .gdx file.
  • Invoke GAMS to run some (any) .gms code which in turn:
    • Reads that GDX 'input' file,
    • Does something—like solving an optimization problem, but not necessarily—and
    • Writes an 'output' file, also in GDX format.
  • If the .gms program runs successfully, update the Scenario with data from that 'output' file.
• In message_ix itself, the class MESSAGE is a subclass of GAMSModel that specifically runs the MESSAGE .gms code.
• See DantzigModel for a minimal, alternate example that uses different .gms sources than the MESSAGE ones.
• GAMSModel is an implementation of the base.Model abstract class.

This architecture was explicitly designed to allow different subclasses of base.Model that use neither GAMS nor the MESSAGE .gms code. These would be arranged as follows:

1. In ixmp, 1+ distinct subclass(es) of base.Model (like PyomoModel, LinopyModel, GAMSpyModel, etc.) with generic features for interacting with model definition code and solvers (i.e. software other than GAMS/.gms/.gdx). For one abortive example, see iiasa/ixmp#420 (and note the date!)
2. In message_ix, 1+ distinct subclass like MESSAGE_Linopy, MESSAGE_GAMSpy that use the corresponding/generic ixmp classes to implement the MESSAGE mathematical formulation.

Would all the GAMS files found in the message_ix folder need to be re-written?

Yes, this would be part of (2).

Is IIASA still considering GAMSpy, Pyomo, or any other alternative to the classic GAMS engine for modernizing the MESSAGE-ix framework?

We are 'considering' only to the extent of keeping the current issue open for discussion. However, to be super clear, there are no resources currently allocated to planning/design/implementation/rollout/support; IOW it is not scheduled/on any concrete roadmap.

Rather, in the short-/medium term we are (and have been for about 12 months) focused on removing a different proprietary software dependency: our central/shared databases for message_ix/ixmp Scenarios are currently on Oracle ($) database servers accessed through Java backend code that is no longer being developed or maintained. (For details of that effort see iiasa/message_ix#939.)

We'd of course welcome community contributions towards the current issue, but I would caution that (a) the time frame, even with 1 FTE/100% effort from a capable developer, would likely be several months to more than a year, and (b) a lot of careful planning before step (1) would be necessary to ensure that step (2) did not run into limitations of the chosen software. For example, see PyPSA/linopy#248: linopy, while seeming 'modern' and enticing, may well fail to cope with MESSAGE-sized LPs and it appears that, instead of relaxing that restriction and improving the features/maturity of the package, people are creating new ones.

[khaeru]

We use CBC as the solver for MESSAGE-ix through the auxiliary_settings.gms file in GAMS. I believe that all the steps in the workflow are handled by and through GAMS, except for the optimization, where CBC is called to optimize the problem and return the solution to GAMS. I'm not aware of any examples of this kind of solver implementation available in GitHub.

I'm not aware either, but this is interesting!

Following my comment above, it would be great to see a PR that extends ixmp.model.gams.GAMSModel to take a "solver" parameter and passes that on to be used by whichever .gms code is invoked. This could be demonstrated/tested using DantzigModel. Then it would be possible to select CBC (or perhaps other solvers, like HiGHS?) through the message_ix Python API, without editing auxiliary_settings.gms.

[criscfer]

(...) it would be great to see a PR that extends ixmp.model.gams.GAMSModel to take a "solver" parameter and passes that on to be used by whichever .gms code is invoked. This could be demonstrated/tested using DantzigModel. Then it would be possible to select CBC (or perhaps other solvers, like HiGHS?) through the message_ix Python API, without editing auxiliary_settings.gms.

I agree! In addition, allowing the user to control some of the solver parameters, like seeds, maximum solutions reached, mip tolerance, etc. would further boost the Python API for message_ix and serve as great advertisement for developers and researchers trying to use the model.