Memory leak in CyClpSimplex

[sphynx]

I'm solving multiple LP problems stored in .mps files. Each problem has around 2000 variables and 1000 constraints, .mps file size is about 9 Mb. I've noticed that the memory is not released after CyClpSimplex goes out of scope.

Here is the minimal example to reproduce:

import glob
from cylp.cy.CyClpSimplex import CyClpSimplex

for mps in glob.glob("./*.mps"):
    CyClpSimplex().readMps(mps)

This leaks with about 4 Mb per problem. I.e. if I run this code on a directory with 3000 MPS files, the memory consumption goes up to about 12 Gb, even though I'm not even solving anything (and not even referring to CyClpSimplex from any variable)

This code which reuses the simplex object doesn't leak:

import glob
from cylp.cy.CyClpSimplex import CyClpSimplex

c = CyClpSimplex()
for mps in glob.glob("./*.mps"):
    c.readMps(mps)

I don't think that actual .mps files matter here, one can probably just use any MPS file of about the size I mentioned and just produce a couple of thousand of copies to reproduce the problem.

It looks like the workaround is easy: just reuse the CyClpSimplex object. But in reality, I'm not loading MPS files but building my models with addVariable and addConstraint, so I have to clean that model at the start of every loop iteration and I'm not sure how to do that reliably, without affecting future solutions.

Could you please either fix this issue or suggest a reliable workaround?

Thank you!

[tkralphs]

There were two recent PRs to address memory leaks (#97 and #101). They are both merged, but the changes are not in a release (I'm waiting for a few other things to be resolved first). Could you try building from source with the current master branch and see if that fixes things for you?

[sphynx]

Thank you for the prompt reply! Do you have a rough estimate as to when the new version with the fix will be released?

I can try building from source, but I don't see any instructions about how to build cylp itself from source (only cbc). Also, it's more complicated than usual becuse I'm behind a proxy and I have to adjust coinbrew to work around that, so I would prefer to avoid the build process if possible.

[tkralphs]

I'm hoping to have a new release within a couple of weeks, but installing from source shouldn't be any more difficult than installing from Pypi. As long as you have git, you can just do

pip install git+https://github.com/coin-or/CyLP

If that fails for some reason because of the proxy, you can always download the code separately (using git clone or by getting a ZIP file from Github) and then just do

pip install .

in the directory with setup.py. All pip install is actually doing is just downloading the code from Pypi and then running pip install locally.

[sphynx]

I see, thanks for the instructions!

There is certain workaround for this memory leak even for the current 0.9.4 version with the leak: one can just create a single CyClpSimplex object and then repeatedly load different CyLPModels into it with simplex.loadFromCyLPModel(model). Something like this:

c = CyClpSimplex()
for _ in range(10000):
    model = CyLPModel()

    # setup model
    model.addVariable(...)
    model += some_constraint
    # ...

   c.loadFromCyLPModel(model)

This doesn't leak because CyLPModel apparently frees its memory correctly (and CyClpSimplex doesn't), so we create just one simplex and many models.

[sphynx]

Hi @tkralphs!

Do you have any news about the release with the leak fix?

I also have a question about my workaround above: is that possible that loadFromCyLPModel(model) does not fully replace the previously loaded model? I.e. is there a chance that this system has actually become stateful when I reuse the same CyClpSimplex?

I'm asking because I seem to observe a weird bug when the problem can't be solved when it's run in a series of other jobs. But when I dump it to .mps file on failure and then load that -- it's perfectly solvable (both via Python API and clp command line tool). I thought, that it might be caused by some statefulness in my current setup and the fact that I reuse CyClpSimplex between calls to loadFromCyLPModel.

Thank you!

[tkralphs]

Thanks for checking. I'm still in the middle of the re-factoring for Cbc, which I hoped to finish before coming back to CyLP. It's turned out to be a rather large effort, so a couple of weeks may have been optimistic :). If you need a release urgently, I can try to make one just to get the leak fix out.

As far as your question, it looks like you do have to be a little careful when re-using CyClpSImplex. What solve method are you using? I would guess that you will be fine if you use, e.g., initialSolve() or initialPrimalSolve() rather than primal() or dual(). That will force a re-solve from scratch, whereas the others may try to do some warm-starting, which may be what's causing the problem. I didn't poke too deeply at this, let me know if this helps.

[sphynx]

Thanks for your reply! Indeed, I am not using initial methods, which may explain what I am seeing.

If you need a release urgently, I can try to make one just to get the leak fix out.

That would be great! I think having a release without the leak will be a very useful thing for everyone.

[bronsonasimov]

This issue is quite old, but I have run into memory leak issues with CyClpSImplex, are there any updates with regards to the memory leak issue?