osqp_benchmarks

osqp_benchmarks copied to clipboard

I ran the Maros Mezaros instances and some QPLIB (those that did not return errors). What I observed is that many objective values do not match those that are reported elsewhere, for example PRIMALC2, DUALC2, BOYD2, PRIMAC5 (MM) and 8792, 8559, 8567 (QPLIB), ... These examples are completely off and not just a little, so it cannot be accuracy.

My suspicion is that there is a problem with the instance parsers, for two reasons: (1) MOSEK returns the same (incorrect) objective values, and (2) QPLIB returns error messages for some problems that upper bound values exceed lower bound values.

I (dimly) recall that we had the same issue with optimal obective values not matching those reported elsewhere, but I can't recall whether the conclusion was that there was an issue with 1) disagreements about the problem data that should be used in the Maros problems or 2) possible errors in one or more places in the objectives reported in some sources.

I think @bstellato eventually resolved the matter - pinging him for an answer.

Take a look at the log from Mittelmann's benchmarks for http://plato.asu.edu/ftp/qpbench_logs/MISC/BOYD2.osqp which gives a different result.

I also looked at the QPLIB problems and could parse and run some after adding a hack that enforces l<=u, which is another reason why I think that there's something going on with the parser.

Thanks for reporting this @loehndorf! There are a few things to comment on here:

• @goulart-paul is right. We had an issue with the Maros parser a while ago. However, it was a minor mistake because we were not adding the objective value constant while reporting the final objective. See https://github.com/osqp/osqp_benchmarks/blob/5c79e870c4bd697383f66f5dff26aea29dc1ebfa/maros_meszaros_problems/maros_meszaros_problem.py#L138-L144

• Maros problems are very badly scaled and a minor change in termination tolerance can cause huge changes in the objective value. That's why you see that behavior in OSQP. We found that the most accurate solver for the objective value on these problems is Gurobi. Have you tried to use that instead of Mosek?

• QPLIB parser might have some issues and we did not extensively test it. That's why in the OSQP paper (and in the README of this repo) we do not report such tests. There might be some fixes needed here.