Resolve dependency issues in setup.py.

[bashir2]

trafficstars

Currently we get some dependency conflicts after installing setup.py and that is because pip cannot resolve version dependencies properly. IOW, instead of picking the intersection of version specifiers between different dependencies, it seems pip installs packages one by one independently (see PR #63 for an example). After installation, pipdeptree shows these issues, for example:

$ pipdeptree -p apache-beam

We need to eventually fix this problem either by adding other non-direct dependencies explicitly (which PR #63 does but is ugly) or finding a tool that can resolve all version restrictions together.

[bashir2]

Bumping the priority of this issue because it keeps haunting us. Today, I was trying to reproduce an issue that broke the build (sample failure) but couldn't do it in my local environment until I deleted/recreated my virtualenv directory. It turned out the issue is beam version 2.5.0 but pip install --upgrade . kept the 2.4.0 in my old environment (and in either case, it produced some version conflict errors which I think is the main source of the dependency version inconsistency).

Basically we need to make sure that pip install --upgrade . does not produce any warnings/errors and then after that make sure that we have a check that can catch dependency regressions.

[bashir2]

To correct a wrong conjecture in my previous message: I think the main reason for not upgrading some packages is that they are not needed by the requirements. When doing pip install --upgrade . there are a lot of lines like this:

Requirement not upgraded as not directly required: google-api-python-client>=1.6 in ./venv/lib/python2.7/site-packages (from gcp-variant-transforms==0.4.0) (1.7.3)

So in this example it kept version 1.7.3 although version 1.7.4 is available. The reason is that it is not necessarily needed based on the requirements. But if I do pip install --upgrade . in a clean virtualenv directory, it picks the latest version, i.e., 1.7.4 in the above example.

[samanvp]

httplib2 incompatibility

• google-api-python-client version 1.7.12 depends on httplib2 0.17.0
• apache-beam version 2.18.0 depends on httplib2<=0.12.0,>=0.8
• Following error occurs:

pkg_resources.ContextualVersionConflict: (httplib2 0.17.0 (/home/travis/build/googlegenomics/gcp-variant-transforms/.eggs/httplib2-0.17.0-py2.7.egg), Requirement.parse('httplib2<=0.12.0,>=0.8'), set(['apache-beam']))
Coverage.py warning: No data was collected. (no-data-collected)

• Solution: pin 'google-api-python-client>=1.6,<1.7.12' in PR #564
• Future AI: revisit this limitation when new versions of Beam are released.