Expose setuptools and wheel version flags for use with non-vendored Pips.

[jsirois]

Right now these are hard-coded but it would be good to allow experimentation with these by end users: https://github.com/pantsbuild/pex/blob/63592214c569c03b60c59766f96fdd2832e698b6/pex/pip/version.py#L77-L86

Pex does not rely on them in any direct way; so a user should be able to try out / pick up a bug fix in either or even downgrade when prodlems are discovered but not fixed.

[danxmoran]

@jsirois I just bumped up against a need for this - I can work on a patch for it.

In the final state, do you want the default versions of setuptools and wheel to still vary based on the requested pip version?

[jsirois]

@danxmoran yes, the default versions should ship statically tied to each Pip release Pex supports. Do you mind documenting the particular case you ran into? Is there a setuptools fix you need for example?

[danxmoran]

Our case is sad - some of our bioinformatics code depends on the (seemingly abandoned) pyvcf package, which still specifies use_2to3 in its setup.py. Support for use_2to3 was dropped in setuptools 58, so we'd like to upgrade to a newer pip while keeping setuptools pinned at an older version.

[jsirois]

@danxmoran Ok. Option #18 is to just fork https://github.com/jamescasbon/PyVCF, use a VCS requirement and plop down a pyproject.toml of:

[build-system]
requires = ["setuptools<58"]
build-backend = "setuptools.build_meta.__legacy__"

[danxmoran]

@jsirois as part of this, should we add setuptools_version and wheel_version to the lockfile schema? I've been working on threading the params through the code and just landed on Lockfile.create

[jsirois]

@danxmoran I'm not sure. Can you supply a backtrace for the error you hit? (You may need PEX_VERBOSE=1 to expose this depending where you hit the error)

It's unclear to me exactly where the surgery is needed and there is a plan to do better for locks (include a sub-lock for every sdist that pins its build requirements). Also, can you confirm the full range of use cases you have. Is it just the class of sdists with 2to3 where you want to use new Pip, old setuptools & wheel or are there other cases?

[danxmoran]

@jsirois no error - I was trying my hand at implementing this by plumbing setuptools_version and wheel_version parameters through the code alongside the existing pip_version.

It's just this one use-case. Noted that we could work around the use_2to3 by forking the repository, we're just averse to it because we've ended up forgetting about / not maintaining the forks when we've taken that approach in the past.

EDIT: I found another case of use_2to3 in our dependency tree, this time in a transitive dependency of a package that we can't upgrade (for Reasons). I don't think the fork+VCS-req approach would save us from this new case.

[jsirois]

In the meantime, --extra-pip-requirement was added in #2461 which just about solves this. The Pip installation module just needs to special case extra_requirements whose project name is setuptools or wheel and let these trump the requirements baked into the given Pip version selected.