pip-upgrader
pip-upgrader copied to clipboard
simion
Crashes if pip-upgrader exists in requirements.txt
% pip-upgrader
<snip>
Traceback (most recent call last):
File "/home/username/.virtualenvs/venv/bin/pip-upgrade", line 11, in <module>
sys.exit(main())
File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/cli.py", line 70, in main
packages, options.get('--use-default-index')).detect_available_upgrades(options)
File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 126, in detect_available_upgrades
package_status, reason = self._fetch_index_package_info(package_name, current_version)
File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 167, in _fetch_index_package_info
return self._parse_simple_html_package_info(package_name, current_version, response)
File "/home/username/.virtualenvs/venv/lib/python3.5/site-packages/pip_upgrader/packages_status_detector.py", line 228, in _parse_simple_html_package_info
latest_version = max([vers for vers in all_versions if not vers.is_prerelease and not vers.is_postrelease])
ValueError: max() arg is an empty sequence
% pip-upgrade --version
1.4.0
% python --version
Python 3.5.3
You might not want to support running pip-upgrader on itself, but I don't think the app should crash because someone has put it in requirements.txt.
I have it in the requirements and it works. Please paste yoyr requirements.txt here or email it to me at [email protected] and i will check which package crashes. Also, this is a good opportunity to catch all upgrade errors for a single package, print them to console and continue upgrading with the rest of the packages. This way, if any other strange package is added, you won't be blocked.
Ah, the problem was that I generated the file with pip freeze > requirements.txt, which locked pip-upgrader to a specific version.
requirements.txt:
...
pip-upgrader==1.4.0
...
crashes.
requirements.txt:
...
pip-upgrader
...
works.
Could you reproduce the bug? For some reason
[vers for vers in all_versions if not vers.is_prerelease and not vers.is_postrelease]
produces an empty list. Adding a guard clause before running max() on it should work, but perhaps the underlying issue is triggered by the filtering done on the previous lines.
I can't reproduce it on my laptop at home either :/
I.e., both versions worked fine with pip-upgrader==1.4.0 in requirements.txt. I will see if I can find the root cause once I get back to work on Monday.
Cool, thanks. If it breaks at work, it should print the problematic package and continue.
For this, you'll have to upgrade pip-upgrader first.
pip-upgrade -p pip-upgrader
The problematic package is pip-upgrade, in my venv at work. I've done some digging and it seems the line
versions_match = re.findall(pattern, response.content.decode('utf-8'), flags=re.IGNORECASE)
produces an empty list for the following response.content string
<html><head><title>Simple Index</title><meta name="api-version" value="2" /></head><body>
<a href="../../packages/e7/1c/5621b74ed130a6c6331865d4e05c6be69e3a703dd8a2a6e672af15156120/pip_upgrader-0.1.0-py2.py3-none-any.whl#md5=13cd80dd4e0abfbf711626cdef323b08" rel="internal" >pip_upgrader-0.1.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/7f/95/2a8f3fbad56f0c044fab9d196b4d53906c892239223d0c0d271561993305/pip_upgrader-0.1.0.tar.gz#md5=9c9303e83734d99dc692a496217296a8" rel="internal" >pip_upgrader-0.1.0.tar.gz</a><br/>
<a href="../../packages/8b/29/2b7839745e5c977fe3644cb2ac0c98e1bec546677587cfbaa96951dd1eb7/pip_upgrader-0.1.1-py2.py3-none-any.whl#md5=b97e169e6b9c03b222837a98655ec9f2" rel="internal" >pip_upgrader-0.1.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/30/a7/88ca8de3f2cbb2a3f53d6bb7d5c83d470eb205c0900f104c420be0911e0f/pip_upgrader-0.1.1.tar.gz#md5=e2b386ea95010de240625c4b63a94ae5" rel="internal" >pip_upgrader-0.1.1.tar.gz</a><br/>
<a href="../../packages/18/86/c34e458362282dfcd704c70d54f4432dc84779b706c29a4413128aa8736b/pip_upgrader-1.0.0-py2.py3-none-any.whl#md5=cd43d3fe7e5ef64e0a084e086360b87f" rel="internal" >pip_upgrader-1.0.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a8/4c/1b0fd5da99ff288f9954c8abc5ea0b08496bbdf361619fd15ae6cdd004bd/pip_upgrader-1.0.0.tar.gz#md5=75b70da482de63fc6dd0fe9e5fabc3f8" rel="internal" >pip_upgrader-1.0.0.tar.gz</a><br/>
<a href="../../packages/4b/a2/bd86365aff1f1ab5c1c47bc5f0311b3a94d151395314cc192acb80918a82/pip_upgrader-1.0.1-py2.py3-none-any.whl#md5=ce7eec58c7cde9f9c708c74ebd77447a" rel="internal" >pip_upgrader-1.0.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a2/e9/8cc706ef5f97ebbaed3074f1eb87bd453ec6e03953135f7d8887243ee711/pip_upgrader-1.0.1.tar.gz#md5=fb3ee8cb0c214e19add0e017b08864d0" rel="internal" >pip_upgrader-1.0.1.tar.gz</a><br/>
<a href="../../packages/d5/54/73e845a3fd5eaa886bb514420e49c819e06b6dde965fd44640c48f5fe014/pip_upgrader-1.1.0-py2.py3-none-any.whl#md5=6c954879e2bc68bcd3291fbef0041f7c" rel="internal" >pip_upgrader-1.1.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/33/bd/f89d891fdb4cf0bd1ea9403cfd50735f8d51d89d4fbbea122fa8b7771990/pip_upgrader-1.1.0.tar.gz#md5=2fcb2ffa4b681ce99e4779ae5abe0d20" rel="internal" >pip_upgrader-1.1.0.tar.gz</a><br/>
<a href="../../packages/69/4f/b0fa5f234b7be7a9bfb081713101e50db8f9353a7b6aec81601f65751b0d/pip_upgrader-1.2.0-py2.py3-none-any.whl#md5=27a85b7027de08fbdf19dfc8f97a45f3" rel="internal" >pip_upgrader-1.2.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/82/03/e8992bb500bdd9e228b898315fde69b6e93ef6182a54b89deb5a908d6985/pip_upgrader-1.2.0.tar.gz#md5=e7da3594fc6f79a65ea09186a71c411e" rel="internal" >pip_upgrader-1.2.0.tar.gz</a><br/>
<a href="../../packages/87/b7/47293388a134d651bfb00983a7ac6bd57093f6513dc77a1d0b4abb9761cf/pip_upgrader-1.3.0-py2.py3-none-any.whl#md5=27fb3f805ec291336379817192c673e0" rel="internal" >pip_upgrader-1.3.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/5c/ef/e00b75c4d933ddfd80319f788f3a266a96f474983e237ae150cd234fcdc0/pip_upgrader-1.3.0.tar.gz#md5=c1c0c8388a3419859fb7b152adfd005b" rel="internal" >pip_upgrader-1.3.0.tar.gz</a><br/>
<a href="../../packages/c4/b2/f4b3cf8b93ea34972a181855026021ae7ab41985280c8873baa0cc31965f/pip_upgrader-1.3.3-py2.py3-none-any.whl#md5=6a72e786c29e782b64b6d3b90554df4e" rel="internal" >pip_upgrader-1.3.3-py2.py3-none-any.whl</a><br/>
<a href="../../packages/90/b9/32bc955ab0b9ef7287db5d3422baac0019ec1a3023dd84506c9aa21524a1/pip_upgrader-1.3.3.tar.gz#md5=7f31f9f1184eceac90d65d3134d877cd" rel="internal" >pip_upgrader-1.3.3.tar.gz</a><br/>
<a href="../../packages/62/f7/4d9609f1b4b98d7f4e7edaabc57e0d812985c198f8c90c45c46197871067/pip_upgrader-1.3.5-py2.py3-none-any.whl#md5=ab83e627298a0a16f4ade8b0aa4024c2" rel="internal" >pip_upgrader-1.3.5-py2.py3-none-any.whl</a><br/>
<a href="../../packages/24/f2/4f6262ccd26692cda30e41aa40f44d338fd7f6744a699260ae62692bd99f/pip_upgrader-1.3.5.tar.gz#md5=f1c329fdcc5f3129dd4b2306c6615ffd" rel="internal" >pip_upgrader-1.3.5.tar.gz</a><br/>
<a href="../../packages/90/41/55fd034f13780c9ced6220e23a8b286b8577e77c323cfc7502c39d210525/pip_upgrader-1.4.0-py2.py3-none-any.whl#md5=ab7d4df56b1dc98dd00926793f227511" rel="internal" >pip_upgrader-1.4.0-py2.py3-none-any.whl</a><br/>
<a href="../../packages/a2/aa/bbec7df19a29c2d3ded829c136116e50d0c2f86287b24fb4162aa9521db8/pip_upgrader-1.4.0.tar.gz#md5=1b401ef39ba8df949427fc422518bd55" rel="internal" >pip_upgrader-1.4.0.tar.gz</a><br/>
<a href="../../packages/1f/44/9135c9e4434ffd057e4b5fbed7af57dafd2dcbcc801322f50a2e85d83fe5/pip_upgrader-1.4.1-py2.py3-none-any.whl#md5=ecc1c675db23044b9ba6c9891ed30ad3" rel="internal" >pip_upgrader-1.4.1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/62/aa/024a4613dca987ca5b2fc00b99343893ccb40e07478945162b4992b81917/pip_upgrader-1.4.1.tar.gz#md5=71e46108739d6e7a15d91844707e4edf" rel="internal" >pip_upgrader-1.4.1.tar.gz</a><br/>
<a href="../../packages/8d/dc/db9b7612d7df5805c440a5a01c66d902e3f8e0f4ed884d2da0b46099a10e/pip_upgrader-1.4.dev1-py2.py3-none-any.whl#md5=63df490e7dd62c6fb1e2257c70519832" rel="internal" >pip_upgrader-1.4.dev1-py2.py3-none-any.whl</a><br/>
<a href="../../packages/0b/84/daab498858f39b0ddbde46c1744137c53c4b78ccc0a52ad0d091acd096c2/pip_upgrader-1.4.dev1.tar.gz#md5=4c8a906b77315fda56b42cd2effdc5b7" rel="internal" >pip_upgrader-1.4.dev1.tar.gz</a><br/>
</body></html>
I'm no regex guru, but I think the problem is that the html blob only refers to pip_upgrader, when the package_name variable is pip-upgrader. I.e., one has a dash, the other an underscore.
Here's my requirement.txt
apipkg==1.4
astroid==1.5.3
certifi==2017.4.17
chardet==3.0.4
colorclass==2.2.0
coverage==4.4.1
decorator==4.0.11
docopt==0.6.2
execnet==1.4.1
idna==2.5
isort==4.2.15
lazy-object-proxy==1.3.1
mccabe==0.6.1
packaging==16.8
pbr==3.0.1
pep8==1.7.0
pika==0.10.0
pip-upgrader==1.4.1
py==1.4.34
pygerrit2==2.0.2
pylint==1.7.1
pylint-mccabe==0.1.3
pyparsing==2.2.0
pytest==3.1.2
pytest-cache==1.0
pytest-cov==2.5.1
pytest-datadir-ng==1.1.0
pytest-mccabe==0.1
pytest-pep8==1.0.6
pytest-pylint==0.7.1
requests==2.18.1
requests-mock==1.3.0
retry==0.9.2
six==1.10.0
terminaltables==3.1.0
urllib3==1.21.1
wrapt==1.10.10
I can reproduce it by running the following commands
$ mkvirtualenv foo
<snip>
$ pip install -r requirements.txt # the requirements.txt from above
<snip>
$ pip-upgrade
Found valid requirements file(s):
requirements.txt
Setting API url to https://artifacts.domain.com/artifactory/api/pypi/all-pypi/simple/{package} as found in /home/username/.config/pip/pip.conf. Use --default-index-url to use pypi default index
1/37: apipkg ... up to date: 1.4
2/37: astroid ... up to date: 1.5.3
3/37: certifi ... up to date: 2017.4.17
4/37: chardet ... up to date: 3.0.4
5/37: colorclass ... up to date: 2.2.0
6/37: coverage ... up to date: 4.4.1
7/37: decorator ... up to date: 4.0.11
8/37: docopt ... up to date: 0.6.2
9/37: execnet ... up to date: 1.4.1
10/37: idna ... up to date: 2.5
11/37: isort ... up to date: 4.2.15
12/37: lazy-object-proxy ... up to date: 1.3.1
13/37: mccabe ... up to date: 0.6.1
14/37: packaging ... up to date: 16.8
15/37: pbr ... up to date: 3.0.1
16/37: pep8 ... up to date: 1.7.0
17/37: pika ... up to date: 0.10.0
error while parsing version
19/37: py ... up to date: 1.4.34
20/37: pygerrit2 ... up to date: 2.0.2
21/37: pylint ... up to date: 1.7.1
22/37: pylint-mccabe ... up to date: 0.1.3
23/37: pyparsing ... up to date: 2.2.0
24/37: pytest ... up to date: 3.1.2
25/37: pytest-cache ... up to date: 1.0
26/37: pytest-cov ... up to date: 2.5.1
27/37: pytest-datadir-ng ... up to date: 1.1.0
28/37: pytest-mccabe ... up to date: 0.1
29/37: pytest-pep8 ... up to date: 1.0.6
30/37: pytest-pylint ... up to date: 0.7.1
31/37: requests ... up to date: 2.18.1
32/37: requests-mock ... up to date: 1.3.0
33/37: retry ... up to date: 0.9.2
34/37: six ... up to date: 1.10.0
35/37: terminaltables ... up to date: 3.1.0
36/37: urllib3 ... up to date: 1.21.1
37/37: wrapt ... up to date: 1.10.10
All packages are up-to-date.
Upgrade interrupted.
$ python -V
Python 3.5.3
$ pip-upgrade --version
1.4.1
I think it's because at work you have a custom package index (artifact) configured in ~/.config/pip/pip.conf.
After upgrading pip-upgrade (use --default-index-url if you still have errors), does it work? It should print the error in console, if any.
Yes, you are correct; that worked with no issue.
At work we should generally go through the custom index (which should pull requested packages from PyPI if they don't exist). In other words, do you have any idea why the custom index might cause the issue?
If not, that's OK. The workaround is simple enough that I can use it instead. But a fix would be nice, because unless it's some kind of misconfiguration of the index on our side, others might encounter the problem as well.
You're right, right now the upgrader does not fallback to pypi when a package is not found.
To be fixed in next release 😉
The solution for this issue is not complicated, but I need to install artifactory to test it properly :)
Do you know what Artifactory is? It's a commercial paid for product. I'm not sure you need Artifactory to test the issue, but there's a free trial here: https://www.jfrog.com/artifactory/free-trial/
Maybe a regular private package index would be enough to test it, without having to deal with Artifactory.