HiGHS_CMD update

[samiit]

Reference issue

553

What does this implement?

An interface for using the HiGHS solver for MILP problems, with a modification to an earlier PR.

Any other comments

Requirements:

We would need to get the necessary binaries for the HiGHS solver from here. The Windows binaries e.g., are here. Usage:

We can now create a HiGHS_CMD object, just like for other solvers, like, Cbc

import pulp as pl
highs_solver = pl.HiGHS_CMD(path = "<file location of HiGHS solver binary>", timeLimit=60, msg=True)
prob = pl.LpProblem("Test")

...

prob.solve(highs_solver)

[jurasofish]

We would need to get the necessary binaries for the HiGHS solver from here. The Windows binaries e.g., are here.

Looks like scipy comes with highs binaries bundled - super quick and easy for any Python user to install. Would it be possible to use them?

see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linprog.html https://docs.scipy.org/doc/scipy/reference/optimize.linprog-highs.html

[CROdominik]

Hi everyone. This issue is linked to the issue 477, but what is the difference? I can see that #477 was merged to the master; however, pl.HiGHS_CMD was not recognized as an atribute even though I downloaded the latest version from the master. Could you explaing me what is the issue with that? thanks a lot!

[samiit]

Hi everyone. This issue is linked to the issue 477, but what is the difference? I can see that #477 was merged to the master; however, pl.HiGHS_CMD was not recognized as an atribute even though I downloaded the latest version from the master. Could you explaing me what is the issue with that? thanks a lot!

Since that issue was already merged, I had to raise a new PR, in order to change the pulp.HiGHS_CMD interface to adjust to the latest HiGHS output file format. @jajhall has shared now the latest output solution format and I am in the process of updating the code in accordance with this. Should be done by this weekend. I checked your other issue with HiGHS_CMD not being recognized as an attribute, but am not able to reproduce it. Can you try with the latest master?

Sam

[CROdominik]

Hi everyone. This issue is linked to the issue 477, but what is the difference? I can see that #477 was merged to the master; however, pl.HiGHS_CMD was not recognized as an atribute even though I downloaded the latest version from the master. Could you explaing me what is the issue with that? thanks a lot!

Since that issue was already merged, I had to raise a new PR, in order to change the pulp.HiGHS_CMD interface to adjust to the latest HiGHS output file format. @jajhall has shared now the latest output solution format and I am in the process of updating the code in accordance with this. Should be done by this weekend. I checked your other issue with HiGHS_CMD not being recognized as an attribute, but am not able to reproduce it. Can you try with the latest master?

Sam

Thanks for the clarification. And I now repeated the process and cloned from the latest master and there is no more the error I mentioned. Thanks for solving it and for this work with the implementation of the solver.

[CROdominik]

Hi @samiit, did you manage to make any progress on this issue? Thanks!

[samiit]

Hi @samiit, did you manage to make any progress on this issue? Thanks!

I am sorry @CROdominik . I was down with COVID but am almost recovered. Will see if I can finish by the coming weekend.

[CROdominik]

Hi @samiit, did you manage to make any progress on this issue? Thanks!

I am sorry @CROdominik . I was down with COVID but am almost recovered. Will see if I can finish by the coming weekend.

Get well soon! And thanks for all the work here.

[iblancoi]

Hi, any update on this issue?

I am trying to run HiGHS via pulp but I am not getting any output. I am using HiGHS for my projects since it's a very powerful solver. It would be awesome to have it fixed soon :)

Thanks in advance, Ignacio.

[chattarinwan]

Hi, thanks for the amazing instruction! I finally managed to use HiGHS with pulp.

However, I did wonder how do you properly input the options = [...] to the solver command? I read some of the HiGHS documents and found that it has "mip_rel_gap" in its option. Then, I tried to input

highs_solver = pulp.HiGHS_CMD(path = <path>, timeLimit=3600, \
                             options=['options.mip_rel_gap=0.15'], msg=True,)

Unfortunately, the solver said PulpSolverError every time I inputted the options list.

Many thanks in advance for your clarification

Cheers, Chattarin W.

[jajhall]

mip_rel_gap is a relatively new option, so maybe it's not in the version of HiGHS you have compiled. Have you got other options to work?

[chattarinwan]

mip_rel_gap is a relatively new option, so maybe it's not in the version of HiGHS you have compiled

Thank @jajhall for the answer. I also tried other options like options =['presolve="on"'] and etc. Still no luck. So I'm wondering if maybe it is because my options inputting format was wrong or sth. Moreover, I use HigHS v1.2.2 which I looked into its source code and did find the existence of "mip_rel_gap" there.

[jajhall]

Thank @jajhall for the answer. I also tried other options like options =['presolve="on"'] and etc. Still no luck. So I'm wondering if maybe it is because my options inputting format was wrong or sth. Moreover, I use HigHS v1.2.2 which I looked into its source code and did find the existence of "mip_rel_gap" there.

Thanks. If this doesn't work, then it may be a bug in the PuLP interface to HiGHS

[chriswasser]

@chattarinwan @jajhall A quick comment on your last discussion about passing options to HiGHS via PuLP: Looking at the code in the current master and the code in this PR, the options argument for HiGHS_CMD only allows for additional CLI options such as --presolve or --solver. This would be achieved by using:

# notice the leading dashes which need to be inserted and the separation between the option and the argument
solver = HiGHS_CMD(options=["--presolve", "on", "--solver", "simplex"])

Passing library options via an option file is currently not supported in PuLP. However, you can modify the write_lines variable in the highs_api.py file of your local installation of PuLP to include any fixed options that always need to be applied for your models such as mip_rel_gap, which would look like this:

# notice the trailing new-line character at the end of each option which is important to produce a valid option file
write_lines = [
    "solution_file = %s\n" % tmpSol,
    "write_solution_to_file = true\n",
    "write_solution_style = 2\n",
    "mip_rel_gap = 0.15\n",
]

[chattarinwan]

@chattarinwan @jajhall A quick comment on your last discussion about passing options to HiGHS via PuLP: Looking at the code in the current master and the code in this PR, the options argument for HiGHS_CMD only allows for additional CLI options such as --presolve or --solver. This would be achieved by using:

# notice the leading dashes which need to be inserted and the separation between the option and the argument
solver = HiGHS_CMD(options=["--presolve", "on", "--solver", "simplex"])

Passing library options via an option file is currently not supported in PuLP. However, you can modify the write_lines variable in the highs_api.py file of your local installation of PuLP to include any fixed options that always need to be applied for your models such as mip_rel_gap, which would look like this:

# notice the trailing new-line character at the end of each option which is important to produce a valid option file
write_lines = [
    "solution_file = %s\n" % tmpSol,
    "write_solution_to_file = true\n",
    "write_solution_style = 2\n",
    "mip_rel_gap = 0.15\n",
]

@chriswasser It works finally! Thanks for your solution.

[jajhall]

Adheres to latest solution file of HiGHS

So I can deprecate the old solution file format that I rescued for you?

[samiit]

Adheres to latest solution file of HiGHS

So I can deprecate the old solution file format that I rescued for you?

Yes. Thanks for the patience. I took really too long, but now managed staying up a bit late!

[samiit]

Hi everyone,

Finally, I managed to get this done. @pchtsp can this be merged now with the main?

Thanks, Sam

[jajhall]

So, am I right in thinking that the Highs MIP solver can now be called from PuLP?

[samiit]

So, am I right in thinking that the Highs MIP solver can now be called from PuLP?

Yes, the only issue is that the user needs to get the latest HiGHS binaries, and needs to create a HiGHS solver like:

import pulp as pl
highs_path = "/path/to/HiGHS/solver"
solver_highs = pl.HiGHS_CMD(path=highs_path)

prob = pl.LpProblem()
...
prob.solve(solver_highs)

[jajhall]

Will it help that HiGHS is now available as pip install highspy (on Linux and MacOS, with Windows WIP)?

Accompanying this will be a pybind11 thin wrapper around each of the methods in the HiGHS C++ class. Proof of concept is done, but we don't yet have all the methods

[samiit]

Just had a look after installing highspy, but could not figure out how this can be used with PuLP yet. I will leave it for @pchtsp @fgenoese or maybe even @stumitchell to comment.

Sam

[samiit]

So, am I right in thinking that the Highs MIP solver can now be called from PuLP?

Yes, the only issue is that the user needs to get the latest HiGHS binaries, and needs to create a HiGHS solver like:

import pulp as pl
highs_path = "/path/to/HiGHS/solver"
solver_highs = pl.HiGHS_CMD(path=highs_path)

prob = pl.LpProblem()
...
prob.solve(solver_highs)

Basically, the binaries are available from the HiGHS_jll.jl folks here

[samiit]

Hi everyone,

Finally, I managed to get this done. @pchtsp can this be merged now with the main?

Thanks, Sam

Sorry @pchtsp , please wait with merge. I noticed that there were a few tests failing with the latest HiGHS binaries.

[samiit]

Hi everyone, Finally, I managed to get this done. @pchtsp can this be merged now with the main? Thanks, Sam

Sorry @pchtsp , please wait with merge. I noticed that there were a few tests failing with the latest HiGHS binaries.

Now all tests corresponding to the HiGHS solver are passing through. Please check and help me in case I missed out anything!

[CROdominik]

@chattarinwan @jajhall A quick comment on your last discussion about passing options to HiGHS via PuLP: Looking at the code in the current master and the code in this PR, the options argument for HiGHS_CMD only allows for additional CLI options such as --presolve or --solver. This would be achieved by using:

# notice the leading dashes which need to be inserted and the separation between the option and the argument
solver = HiGHS_CMD(options=["--presolve", "on", "--solver", "simplex"])

Passing library options via an option file is currently not supported in PuLP. However, you can modify the write_lines variable in the highs_api.py file of your local installation of PuLP to include any fixed options that always need to be applied for your models such as mip_rel_gap, which would look like this:

# notice the trailing new-line character at the end of each option which is important to produce a valid option file
write_lines = [
    "solution_file = %s\n" % tmpSol,
    "write_solution_to_file = true\n",
    "write_solution_style = 2\n",
    "mip_rel_gap = 0.15\n",
]

@chriswasser It works finally! Thanks for your solution.

Hi @chattarinwan , do you know more about when could this be added to the pulp, and how much work around it there is?

Apparently, Highs has many options that can be passed via an option file: https://www.maths.ed.ac.uk/hall/HiGHS/HighsOptions.html

thanks

[CROdominik]

Hi everyone, Finally, I managed to get this done. @pchtsp can this be merged now with the main? Thanks, Sam

Sorry @pchtsp , please wait with merge. I noticed that there were a few tests failing with the latest HiGHS binaries.

Now all tests corresponding to the HiGHS solver are passing through. Please check and help me in case I missed out anything!

Great work @samiit , thank you for your effort!

[chriswasser]

Hi @chattarinwan , do you know more about when could this be added to the pulp, and how much work around it there is?

Apparently, Highs has many options that can be passed via an option file: https://www.maths.ed.ac.uk/hall/HiGHS/HighsOptions.html

thanks

I have been working on my own fork of PuLP since I needed some of the file options too - as well support for threads, logPath, gapRel, gabAbs, and mip. However, I have not been putting extra care in the update to the HiGHS API to remain backwards compatible with Python 2 and <3.7 since they have all reached there end of life (some a really long time ago). Maybe this would be a good time to migrate to a more modern Python version range i.e. >=3.7!? That’s primarily for the maintainers of PuLP to decide though. I’m happy to open a PR with the changes (either as an addition to this existing one or a new one) if desired. In case backwards compatibility is crucial, the non-compatible code parts like f-strings and typehints could be reverted.

Cheers

Christian

[pchtsp]

@chriswasser , please create a new PR with these improvements on the fork. We can help make it Backwards compatible, it will probably be minor things.

@samiit thanks for the work.

[CROdominik]

Will it help that HiGHS is now available as pip install highspy (on Linux and MacOS, with Windows WIP)?

Accompanying this will be a pybind11 thin wrapper around each of the methods in the HiGHS C++ class. Proof of concept is done, but we don't yet have all the methods

HI, does this mean that we can use highs inside pulp by pip install highspy, without pointing to highs binaries?

[samiit]

Will it help that HiGHS is now available as pip install highspy (on Linux and MacOS, with Windows WIP)? Accompanying this will be a pybind11 thin wrapper around each of the methods in the HiGHS C++ class. Proof of concept is done, but we don't yet have all the methods

HI, does this mean that we can use highs inside pulp by pip install highspy, without pointing to highs binaries?

I am not sure about it. Maybe @jajhall or others may comment!