Installing cylp in a robust, portable, neat way

[matt-telstra]

This is following on from Issue #47 (I can't figure out how to reopen that issue)

Intro

I want to use cylp on a system for which pip cannot find precompiled binaries. pip install cylp should still work. It should compile things for me, but it does not.

As a user, I expect to be able to install cylp solely by doing

pip install cylp

Requiring more steps than that is not ideal. Requiring users to manually download and manually compile many dependencies is not user friendly, nor is it robust.

Furthermore, I expect that using pip install cylp with a virtual environment will install cylp and all dependencies into that environment. For context, I'm deploying my code into AWS Lambda functions. That involves zipping up a virtual environment. For every library except cylp, I can just use pip, and zip up the directory containing the virtual environment.

I understand that packaging binaries in python libraries neatly is non-trivial. However it is certainly possible. Numpy and pandas use C files too, and they work seamlessly. I realise I'm asking for a lot by requesting that pip work seamlessly. I realise that this library and it's dependencies are probably written by a one or two people in their spare time. I'm not trying to criticise. I'm trying to help. That's why I've included published 2 scripts which took me several weeks to write. This is what it takes to install cylp in a robust, reliable and reproducible way (on machines for which pip cannot find pre-compiled binaries).

This process involved me figuring out the extensive dependency tree (including dependencies which pip does not know about), and the fragile version incompatibilities (e.g. did you know that there are 2 different versions of numpy both numbered 1.12.1?), and then manually downloading the source code and compiling each of those dependencies in their own custom virtual environment, extracting .whl files from one virtual env and installing it in the next. In short, it is a nightmare.

The scripts

I've split up this script into 2 steps, because the first script takes a very long time, so you don't want to run it every time you rebuild your virtual environment.

• compile.sh compiles all the libraries into .whl files
• makeScript.sh script takes the .whl files compiled by compile.sh, and installs them into a virtual env, in such a way that pip is aware of (almost) all dependencies simultaneously

Details and Outstanding issues

• This script needs to modify the LD_LIBRARY_PATH OS variable. That's messy and crude, and means this script can't easily go into a normal setup.py file.
• This script works on my machine, which is an EC2 instance running Amazon's version of Redhat. When I try running it on a fresh machine, I still get some issues. I'm just posting what I have anyway.

Conclusion

I believe a user-friendly installation process is by far the most important issue facing this library, because the current process makes it near impossible to use.

I'm not sure whether the cause of all these issues is cylp itself, or rather CBC and SCS. I'm posting here first to see what you say.

What’s the best way to add these scripts to setup.py, so that pip does all this for the user? To restate my goal: I want to be able to install cylp and its dependencies into a self-contained virtual environment using a single pip command.

Or failing that, I just hope that other users struggling with the installation process can copy and paste my work here to save themselves weeks of hair pulling and confusion.

[scollis]

Most importantly we need Py3!

[si14]

Hey @tkralphs , do you think you might find a bit of time to check this issue out? Between this and py3 issues it's really a showstopper both for this lib and to the use of CBC with CVXPY.

[tkralphs]

Yes, I'll look at it as soon as I can.

[GiorgioBalestrieri]

@tkralphs great work on making cylp work with Python 3 on Windows. I think making pip install cylp work on Linux (and other OS's, for that matter) is really key to make it usable at production level.

[tkralphs]

I did take a look at these scripts. To understand everything they're doing and make the whole thing bulletproof looks like a good amount of work, so I guess I need to understand better what the need is for something like this. I have no problem myself doing pip install cylp in Linux and I think it should work without a problem for most people, but I realize that the use case described in this issue is an entirely different animal. Are there many people out there with the need for such an installation procedure? When is it not enough to (possibly) install a few packages to get dependencies and then to install cylp? The scripts seem like overkill in all but a few cases.

[scollis]

Hey @tkralphs does the pip command work without setting the COIN_INSTALL env variable (or whatever it was)? That is my biggest issues in using CyLP in reproducible Conda environments

[scollis]

BTW Kudos on the work you have been doing.

[tkralphs]

There has to be some way to find the installation, which was the purpose of that variable, but I've now added several other ways, including checking the PATH for clp and looking for .pc files in the PKG_CONFIG_PATH. If you've installed Cbc with a package manager in Linux or with Homebrew in OSX, you shouldn't need to do anything. I guess this is about as good as it can get. Let me know if there are further improvements I can make for your use case.

[matt-telstra]

When is it not enough to (possibly) install a few packages to get dependencies and then to install cylp?

My use case was running cylp inside an Amazon Lambda function. It took me weeks to get that working.

[tkralphs]

@matt-telstra Yeah, I did get your use case, but don't know how common that is and don't want to spend heaps of time making your solution bullet-proof if there's not much demand. Too many fish to fry :).