Py3 branch can not be installed with setuptools (dependency_links, install_requires), only through requirements.txt, due to version overload

[s-m-e]

In a Python 3 project, I used to install wikitools by adding a line in requirements.txt pointing to the py3 branch looking this this:

-e git+https://github.com/alexz-enwp/wikitools.git@py3#egg=wikitools

This works fine for development purposes. However, for releasing the package, it is recommended to define dependencies in setup.py instead, as Donald Stufft (the core developer of PyPI) explains in this blog post. However, in a nutshell, a setup.py in of the following form will install the Python 2 version from PyPI instead of the Python 3 version from GitHub:

from setuptools import setup
setup(
    dependency_links = [
        'https://github.com/alexz-enwp/wikitools/tarball/py3#egg=wikitools_py3-1.2'
        ],
    install_requires = [
        'wikitools==1.2'
        ]
    )

The problem is that wikitools is already available in version 1.2 (and 1.3) on PyPI, so pip will give it priority. The py3 branch contains something labeled as version 1.2, which will therefore be ignored by pip.

The problem can be solved in three ways:

• The wikitoolsmodule for Python 3 is renamed into something like wikitools_py3. Without starting a new repository, this can be achieved by simply changing the name parameter in wikitools' own setup.py in the existing py3 branch. EDIT: It actually requires a lot of changes, more than I thought, see pull request, so the next option might be a lot easier.
• Alternatively (temporarily) the version parameter in wikitools' own setup.py in the py3 branch can be pumped to something higher than what is available on PyPI, e.g. 1.4 (as of today's latest version on PyPI being 1.3).
• The ideal (theoretical) solution would be if wikitools had a common codebase for supporting both Python 2 and Python 3, which wont be trivial ...

For reference, two related discussions on StackOverflow:

• Force setuptools to require package from repository instead of PyPI
• Setuptools unable to use link from dependency_links

[mzmcbride]

Thanks for researching this.

I think that https://github.com/alexz-enwp/wikitools/pull/51 is inelegant and it doesn't seem like a direction we want to pursue currently.

The 1.3 and 1.4 versions of wikitools are a bit of a mess, as I recall (cf. https://phabricator.wikimedia.org/T102862#2296124). Nobody's had the time or patience to clean them up. (I'm also still not totally sure how new versions of packages get published to PyPI.)

Regarding Python 3 support generally, as a user, do you think maintaining Python 2 compatibility indefinitely is necessary? What do you think about leaving the wikitools 1.x series compatible with Python 2.x and making the wikitools 2.x series compatible with Python 3.x?

[s-m-e]

@mzmcbride I'd agree, the code is in a bit of a messy state - but nevertheless, it actually is very useful. I thought about cleaning it up, but I am lacking a properly working instance of mediawiki for testing. There probably are preconfigured vm or docker images out there ... The test suite of wikitools looks highly promising and worth the effort.

Python 2 support until the end of time, yeah. Well, I am still seeing Python 2.6 setups in the wild (as opposed to 2.7, which would be progress), and I am still migrating code here and there to Python 3, so for now, it does make sense to have a working version for Python 2 around, I'd say. On the other hand, I do not like cross-compatible code (i.e. working with six), because it tends to create all sorts of wired and hard to debug issues. I'd vote for two code bases (in one project) and probably a way of building them into separate wheels for Python 2.x and 3.x.

[s-m-e]

@mzmcbride Cleaning up and refactoring python-wikitools (for use with Python 3)