Parallel scenario run raise during remove cplex.opt a FileNotFoundError

[tzipperle]

I have figured out a new problem for me. I get the following error some times when I run my scenario parallel runner:

 File "D:\Modelle\MESSAGEix-GHD\ghdmod\core.py", line 83, in make_parallel_scenario
    scenario.solve('MESSAGE')
  File "C:\anaconda3\envs\message-ghd-new\lib\site-packages\message_ix\core.py", line 366, in solve
    os.remove(fname)
FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden: 'C:\\anaconda3\\envs\\message-ghd-new\\lib\\site-packages\\message_ix\\model\\cplex.opt'

[khaeru]

@tzipperle thanks for filing the issue. Have you previously shared this "scenario parallel runner"/could you? That would help devs reproduce the issue, create a test, etc.

One thought is that this is related to iiasa/message_ix#141; if ixmp gave every solve() its own temporary directory, then cplex.opt could be created in that directory and be guaranteed protection from parallel execution.

[tzipperle]

Hi @khaeru ,

I think this could work with protected directories. Do you know what happens when I start slove() in a parallel run without clpx.opt?

[tzipperle]

I have did an other test and I got the following error:

*** Error Cannot open parameter file "C:\anaconda3\envs\message-ghd-new\lib\site-packages\message_ix\model\cplex.opt"
*** Error Error code = 2; No such file or directory
Reading data...
Space for names approximately 6.95 Mb
Use option 'names no' to turn use of names off
CPXPARAM_Advance                                 0
CPXPARAM_Simplex_Limits_Iterations               100000000
CPXPARAM_TimeLimit                               1000000
CPXPARAM_Threads                                 1
CPXPARAM_Parallel                                1
CPXPARAM_Tune_TimeLimit                          200000

Do you have a idea how I could fix this? I'm not so sure anymore if I should use version 1.2.0 in my productive environment or not?

[khaeru]

I'm realizing this is actually in message_ix (so perhaps this issue should be moved): https://github.com/iiasa/message_ix/blob/8a4587544b242744da21439af9e61bf986d6733e/message_ix/core.py#L353-L367 …and we must unfortunately blame @gidden :wink:

My suggestion as a temporary fix would be to install from source, and then simply comment out the os.remove(fname) line, and possibly also the with open(fname, 'w') as f: block. This will disable the portions of the code that touch that file.

[tzipperle]

Thanks for your suggestion. Do you know is there a possibility to change the default options in GAMS/CPLEX to the same as now in message-ix? Then I could catch the remove error in my code.

[khaeru]

I'm not sure I understand the question, but if you prevent the code from writing cplex.opt (as in my last comment), and then manually put a file there with your preferred options (maybe make it read only like chmod -w cplex.opt), it should work?

[tzipperle]

My thought was that if the file doesn't exist, GAMS will have the same settings. Currently others are used (eg. Threads=1 and not 4).

My solution now is to exclude the line os.remove() from the conda environment in the site-packages...

[khaeru]

I moved this issue from ixmp to message_ix, as the code still lives in message_ix.

However, as mentioned earlier, I think the fix here should be both here and in ixmp:

One thought is that this is related to #141; if ixmp gave every solve() its own temporary directory, then cplex.opt could be created in that directory and be guaranteed protection from parallel execution.

This would now be handled within the ixmp GAMSModel class. The features for writing the optfile could be moved upstream at the same time.

[khaeru]

Marked as a duplicate of #222, which was reported a day earlier.

[gidden]

closed via https://github.com/iiasa/message_ix/pull/657