Handle package updates that add another subdependency

[edmorley]

pyup-bot's initial PR updated pytest-html from 1.13.0 to 1.14.2.

The new version has apparently added a dependency on pytest-metadata, which means the Travis run failed with:

In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    ...
    pytest-metadata from https://pypi.python.org/packages/71/33/9037033094e0f8da4589ab147bf3d931746f9a322300e705e49c84db5ed1/pytest_metadata-1.3.0-py2.py3-none-any.whl#md5=f9ad72f4d02315c99390f673c5bbd582 (from pytest-html==1.14.2->-r requirements/dev.txt (line 84))

(https://travis-ci.org/mozilla/treeherder/jobs/219388305)

To fix, the new package will need adding to the requirements file, along with an appropriate version number and hash.

This may be beyond the scope of pyup, but filing just in case it's something you'd want to support.

[jayfk]

I'd love to add support for a feature like this, but it would require to resolve transitive dependencies on the server side.

That's something I want to avoid at all costs because it basically means installing each and every dependency somewhere in a sandbox.

[edmorley]

Agreed that is undesirable.

I was going to say how about using the PyPI API, eg: https://pypi.python.org/pypi/pytest-html/1.13.0/json -> ['info']['requires_dist'] = ["pytest-metadata", "pytest (>=2.3)"] https://pypi.python.org/pypi/pytest-html/1.14.2/json -> ['info']['requires_dist'] = ["pytest (>=2.3)"]

However it appears not all packages have those attributes populated, eg: https://pypi.python.org/pypi/celery/json

[jayfk]

Maybe we can use libraries.io to resolve dependent projects: https://libraries.io/pypi/requests/dependents

Could you jump in here @andrew? How do you resolve transitive dependencies on libraries.io for Python packages?

[jayfk]

There's also a GSoC project: https://mail.python.org/pipermail/distutils-sig/2017-March/030202.html

[andrew]

For PyPI dependencies, I have a server that attempts to run pip install foobar==1.0.0 -d and greps out the resulting dependencies install for python 3 on linux. These dependencies are then added to the system in a standard format and I have written a general purpose transitive dependency resolver that works (mostly) for all package managers.

So it's a bit of a hack, we have a JSON API for resolving the trees and listing top level dependencies but it work be perfect for Python as setup.py could do all kinds of conditionals based on the environment it's ran in.

[jayfk]

That's amazing. Thanks for your work on this!

[andrew]

@jayfk here's the python specific bit: https://github.com/librariesio/pydeps was a quick hack, if it's useful could potentially set it up to allow passing more args like python or pip versions and run each python version in a virtual machine.