charm-helpers Remove Makefile (was: ERROR: Could not find a version that satisfies the requirement distro-info)

I don't believe this is an issue of charm-helpers itself, but still make test doesn't pass the phase of setting up the venv as follows. FWIW, I'm on Ubuntu 20.10, not 20.04 LTS.

$ make test
dpkg-query -W -f='${status}' gcc python-dev python-virtualenv 2>/dev/null | grep --invert-match "not-installed" || sudo apt-get install -y python-dev python-virtualenv
install ok installed
virtualenv .venv --system-site-packages
created virtual environment CPython3.8.6.final.0-64 in 87ms
  creator CPython3Posix(dest=/home/nobuto/src/corporate/charm-helpers/.venv, clear=False, global=True)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/nobuto/.local/share/virtualenv)
    added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
.venv/bin/pip install -U pip
Collecting pip
  Using cached pip-21.0.1-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
Successfully installed pip-21.0.1
.venv/bin/pip install -I -r test-requirements.txt
Collecting git+https://git.launchpad.net/ubuntu/+source/python-distutils-extra (from -r test-requirements.txt (line 3))
  Cloning https://git.launchpad.net/ubuntu/+source/python-distutils-extra to /tmp/pip-req-build-84nfnn4z
  Running command git clone -q https://git.launchpad.net/ubuntu/+source/python-distutils-extra /tmp/pip-req-build-84nfnn4z
Ignoring ipaddress: markers 'python_version < "3.0"' don't match your environment
Ignoring PyYAML: markers 'python_version >= "3.0" and python_version <= "3.4"' don't match your environment
Ignoring Jinja2: markers 'python_version >= "3.0" and python_version <= "3.4"' don't match your environment
Ignoring Jinja2: markers 'python_version == "2.7" or python_version == "3.5"' don't match your environment
Collecting pip
  Using cached pip-21.0.1-py3-none-any.whl (1.5 MB)
Collecting coverage>=3.6
  Using cached coverage-5.4-cp38-cp38-manylinux2010_x86_64.whl (245 kB)
Collecting mock<1.1.0,>=1.0.1
  Using cached mock-1.0.1.zip (861 kB)
Collecting nose>=1.3.1
  Using cached nose-1.3.7-py3-none-any.whl (154 kB)
Collecting flake8
  Using cached flake8-3.8.4-py2.py3-none-any.whl (72 kB)
Collecting testtools==0.9.14
  Using cached testtools-0.9.14.tar.gz (146 kB)
Collecting path<14.0.0
  Using cached path-13.3.0-py3-none-any.whl (21 kB)
Collecting amulet
  Using cached amulet-1.21.0.tar.gz (30 kB)
Collecting distro-info
  Using cached distro_info-0.0.0-py3-none-any.whl
WARNING: Discarding https://files.pythonhosted.org/packages/91/e0/ccd731d32238bc18bb8ac80353b4028394f80d5ff7db0f93a5ba488690d3/distro-info-0.10.tar.gz#sha256=3c5c2c800e84f36e56e4c59b09981eae357127494f9eb391712888c3f7ac2916 (from https://pypi.org/simple/distro-info/). Requested distro-info from file:///home/nobuto/.cache/pip/wheels/70/41/44/7433bd9e251aed01842961adf793059948d1de98d25877c5b8/distro_info-0.0.0-py3-none-any.whl (from -r test-requirements.txt (line 13)) has inconsistent version: filename has '0.10', but metadata has '0.0.0'
ERROR: Could not find a version that satisfies the requirement distro-info
ERROR: No matching distribution found for distro-info
make: *** [Makefile:45: .venv] Error 1

Feb 19 '21 03:02 nobuto-m

It's a bug in distro-info due to pip 21 being more picky about consistency in version numbers specified in the package. Affects charm-helpers. Also, pip 21 drops support for py27 which charm-helpers still supports, which means that charm-helpers isn't compatible with pip 21.

Possible steps:

Pin pip < 20.3 in the tests in the short term (that's when the the new resolver became really picky). This solves the 'immediate' problem.
Fix distro-info and then pin pip < 21.0 to retain the py27 compatibility.
Update charm-helpers to remove py27 support (major version release) and then unpip pip again (but say add an upper constraint to < 22 to avoid future major breaking changes).

Thoughts on this approach?

Feb 19 '21 09:02 ajkavanagh

Note that using make is deprecated (really). There is a tox.ini file which pins pip appropriately, and (hopefully) tox -e py3 works?

Feb 19 '21 18:02 ajkavanagh

Note that using make is deprecated (really).

That makes sense now. Getting a deprecate warning or deleting Makefile would be nice though. If I find a Makefile in a new project to me, I always try to see make test first just to learn the convention in the project. And if it doesn't exist, I look into tox.ini.

There is a tox.ini file which pins pip appropriately, and (hopefully) tox -e py3 works?

Yes, it will pass the prep phase and start executing tests. It will fail with some tests though, https://github.com/juju/charm-helpers/issues/575

Feb 22 '21 06:02 nobuto-m

Getting a deprecate warning or deleting Makefile would be nice though.

That's a fair point. We should really get rid of it; I think it's just old cruft now, (although there is a Makefile for sphinx, but we ought to push that to a tox.ini as well, to get it into a virtualenv).

Feb 22 '21 10:02 ajkavanagh