highspy hangs during solve for some models

[SolverMax]

Issue: HiGHS hangs when solving some models using highspy. It occurs more often when the model is running as a JupyterLab notebook, but also occurs if the same model is run from the command line as a py file.

When the solver hangs, CPU usage drops to zero but the memory usage remains.

OS: Windows 11.

HiGHS: 1.7.0 (though same behavior occurred in 1.5.3)

Python: Same behavior on several different versions of Python.

Example model: https://github.com/SolverMax/Random/tree/main/HiGHS-testing/Hangs

The notebook in that folder is an example of a model where this behavior occurs. The HiGHS solver stops after about 50 seconds of runtime.

[Coloquinte]

It's likely related to the thread bug on Windows, #1044 and #1639. You can work around it by setting the threads=1 option

[jajhall]

Thanks @Coloquinte, I concur. Let's relate this to #1044

[SolverMax]

Setting threads=1 does not work. The behavior is the same.

[jajhall]

I need Data-100.json to test this example

[SolverMax]

data-100.json is in the same folder. I've just revised the notebook to refer to data-100.json rather than Data-100.json

[jajhall]

I still can't get the notebook to load the file. I've almost no experience with Colab.

[galabovaa]

I will look at this, thanks! I should also investigate the threads bug.

[galabovaa]

@SolverMax Could you please try our pre-release 1.7.1.dev1 version from PyPI and confirm that you still get the issue? It is likely that you will, but I need it as a sanity check, since I will go from that and quite a lot has changed in the highspy build between 1.7.0 and 1.7.1.dev1

[SolverMax]

The behavior is the same with version 1.7.1.dev1.

I've added hangs.py to https://github.com/SolverMax/Random/tree/main/HiGHS-testing/Hangs Running that file from the command line, HiGHS also hangs at about the 50 second point. You'll probably need to change the paths for the DataFilename (line 9) and the log file (line 72).

[SolverMax]

Update: While attempting to run a different model using the command line, I noted that HiGHS produces a large volume of output when it starts after a previous HiGHS call had been aborted because the solver hung. The output ends with: \\\\n\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\'\\\\\\\\n\\\\\\n\\\'\\\\n\\\'\n\\n\'\n'' Arguments: ()

Suppressing console logging, by setting the log_to_console option to False, allows the solver to run to completion without hanging. Writing to a log file still works, so I can monitor progress that way.

Therefore, this issue of the solver hanging appears to be related to other issues around HiGHS output. I hope this is useful.

Also, sometimes the output includes text like: ERROR: Output stream (<_io.TextIOWrapper name='' mode='w' encoding='utf-8'>) closed before all output was written to it. The following was left in the output buffer: 'ERROR: Output stream (<_io.TextIOWrapper name='' mode='w'\nencoding='utf-8'>) closed before all output was written to it. The following\nwas left in the output buffer: 'ERROR: Output stream (<_io.TextIOWrapper\nname=\'\' mode=\'w\'\nencoding=\'utf-8\'>) closed before all output\nwas written to it. The following\nwas left in the output buffer: \'ERROR:\nOutput stream (<_io.TextIOWrapper\nname=\\\'\\\'\ nmode=\\\'w\\\'\\nencoding=\\\'utf-8\\\'>) closed before all output\ nwas\nwritten to it. The following\\nwas left in the output buffer:\n\\\'ERROR:\nOutput stream (<_io.TextIOWrapper\\nname=\\\\\ \\\'\\\\\\\'\\nnmode=\\\\\\\'w\\\\\\\'\\\\nencoding=\\\\\\\'utf-8\\\\\\\'>) closed before all\noutput\\nwas\nwritten to it. The following\\\\nwas left in the output\nbuffer:\n\\\\\\\'ERROR:\\nOutput stream (<_io.TextIOWrapper \\\\nname=\\\\\\\\\\n\\\\\\'\\\\\\\\\\\\\\\'\\\nnmode=\\\\\\\\\\\\\\\'w\\\\\\\\\\\\\\\'\\\\\n\\\\nencoding=\\\\\\\\\\\\\\\'utf- 8\\\\\\\\\\\\\\\'>) closed before\nall\noutput\\\\nwas\\nwritten to it. The following\\\\\\\\nwas left in the\noutput buffer:\n"ERROR:\\\\nOutput\\nstream (<_io.TextIOWrapper\\\\\\\\ \nname=\\n\\\\\\\\\\\\\\'\\\\\\\\\\\\\\\'\n\\\\nmode=\\\\\\\\\\\\\\\'w\\\\\\\\\\\n\\\\\'\\\\\\\\\\\\\\\\nencodin\\ng=\\\\\\\\\\\\\\\'utf-8\\\\\\\\\\\\\\\'>)\nclosed\nbefore all output was\\\\\\\\nwritten to\nit.\\\\nThe\\nfollowing\\\\\\\\\\\\\\\\nwas left in\nthe output buffer:\n\\\\\\\\\\\\\\\'Running\nHiGHS\\\\\\\\n1.5.3\\n[date:\\\\n2023-05-16,\ngit\\\\\\\\\\\\\\\\nhash:\n594fa5a9d]\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\nCopyright\n(c)\n2023\\nHiGHS\\\\\\\\nunder MIT\\\\nlicence\nterms\\\\\\\\\\\\\\\\\\\\\\\\\n\\\\\\\\n\\\\\\\\\\\\\\\'\\\\\\\\\\\\\\\\n"\\\\\\\\n\\\\\\\'\\\\n\\\'\\n\'\n'\n' WARNING: Significant delay observed waiting to join reader threads, possible output stream deadlock

[jajhall]

Thanks for this. Logging when calling via Python appears to be a bit different.

I'll get back to this in due course, but I'm currently in the few weeks of the year when I'm busy with teaching

[Coloquinte]

Which version of Pyomo are you using? It looks a lot like this Pyomo bug

[SolverMax]

I was using Pyomo 6.7.0, but just upgraded to latest 6.7.1 - same behaviour. Only HiGHS behaves this way. Other solvers (including CBC, GLPK, Gurobi, Couenne, Bonmin, Ipopt) all behave as expected.

[jajhall]

I wonder why no-one else is reporting this behaviour with HiGHS

Adding a flush to stdout and the log file channel before returning from HiGHS methods would be easy to do

Do you get the same issue if you 'pip install highspy' locally? If so, it would make debugging rather more feasible.

[SolverMax]

highspy is installed locally, using pip.

I can do more debugging, if you tell me what to do.

[jajhall]

Good, thanks. I won't be able to give you anything to try for another ten days, though

[Coloquinte]

For the error output, I think it is because the log header is written even during calls to addVars or addRows, but only the output of solve() is intercepted by Pyomo here. That could be a patch for Pyomo (or just no header output for those functions). It should be unrelated to the hang.

[jajhall]

Output of the header must come before any other HiGHS logging. I'd have expected Pyomo to be able to echo logging from any HiGHS method, otherwise error messages will not be displayed. For example, if there is an error in the definition of a variable or constraint, then it would seem important that HiGHS reports the cause to the user. Perhaps Pyomo does some checking in the method that calls addVars or addRows so that such errors cannot occur, but does it catch everything that all solvers will reject?

[Coloquinte]

You're right, it's not probably not a change to make on Highs' side.

Then I think it's a matter of adding the following code around all calls involving self._solver_model in pyomo's highs.py. I can open an issue on their side.

with TeeStream(*ostreams) as t:
            with capture_output(output=t.STDOUT, capture_fd=True):
               ...

[jajhall]

Then I think it's a matter of adding the following code around all calls involving self._solver_model in pyomo's highs.py. I can open an issue on their side.

Thanks. I'm still puzzled that we've not had more people reporting this

[Coloquinte]

Thanks. I'm still puzzled that we've not had more people reporting this

No idea either. Maybe it only happens on Windows? Or everyone saw it and ignored it 😂

[jajhall]

Not a HiGHS issue, so closing this

[Coloquinte]

I think Highs hanging may still be a Highs issue

[SolverMax]

Why is this closed? HiGHS still hangs when solving some models.

[jajhall]

Why is this closed? HiGHS still hangs when solving some models.

Sorry, I thought that not handling the logging was causing HiGHS to hang

[SolverMax]

It might be. Setting the log_to_console option to False stops HiGHS hanging. But that significantly reduces usability.

[Coloquinte]

It might be. Setting the log_to_console option to False stops HiGHS hanging. But that significantly reduces usability.

Then it's likely connected to Pyomo. We'll know once it's fixed there. At least solving the mps works just fine

[jajhall]

I've looked in src/io/HighsIO.cpp, and (currently) any logging to console or a file is being flushed immediately. This may not be the case for all cases of writing models, solutions etc to files, because that's not done through this mechanism, but I don't have time to check all these

[Syuizen]

+1 I have encountered this issue when solving models on windows 11, but never have such problems when using Mac before.

I tried to use subprocess.run to call highs.exe to solve a mps file in python 3.12.2, this should be similar to call highs directly from the command line. However, the solver kept hanging on after "writing solution to ...".

This issue does not happen all the time but pretty frequently (like 80%).

Specs:

• System: Windows 11 Enterprise
• Highs version: 1.7.1
• Python: 3.12.2

Update: I previously used the pre-compiled exe. After recompiling, it seems works well.

[akrishnamo]

I am experiencing a similar issue with Windows 11 and JupyterLab locally. The Kernal dies when i set the ortools solver to HIGHS.

However, HiGHS Works like a charm on ortools on Google Colab, runs superfast compared to SCIP and CBC