Allow to skip downloading manylinux wheels

[sashkab]

• Pip version: 8.1.{0,1,2}
• Python version: 2.7
• Operating System: CentOS 6

Description:

When PIP 8.1 introduced support for manylinux1 wheels, few issues started to show up when attempting to build wheels. We use custom Python installation, installed in other than system Python location. When we upgrade our package requirements, we re-build wheels ourself using command below:

$ pip wheel -w /path/to/wheel -f /path/to/wheel --use-wheel  -r requirements.txt

Pre-PIP 8.1, this command did what I expected: build wheels for new packages in the requirements.txt file. After PIP 8.1, it just download manylinux wheels despite the fact that /path/to/wheel already has an wheel for the requirement.

Let's take an example: One of the packages we use is numpy. Requirements string looks like this: numpy==1.10.4. This ensures we use only this package version. What PIP pre-8.1 did: It detected that wheel for numpy 1.10.4 was already built and did nothing else. What PIP 8.1.x does: it downloads manylinux1 wheel, despite the fact that I already have wheel for 1.10.4 and wasn't updating numpy wheel.

You might suggest adding --no-binary numpy, but that won't solve my problem either -- I don't want to rebuild numpy package every time I build wheels, and I don't want to select only updated packages to build wheels. I like what I had before: -r requirements.txt and it did the job.

So what I'm asking here is either of two:

1. If wheel already built -- just skip it, don't attempt to download manylinux wheel.
2. Option to disable download of manylinux wheels.

[dstufft]

You can drop a _manylinux.py file in site-packages or in the standard library or wherever that will make it importable with contents like:

manylinux1_compatible = False

Does that satisfy your use case?

[sashkab]

Thank you for quick response. It might -- need to test this.

But I'd still prefer some kind of command line option (ie --no-manylinux1) for pip.

[asottile]

Possibly related: http://stackoverflow.com/questions/37231799/exclude-manylinux-wheels-when-downloading-from-pip?noredirect=1#comment62039099_37231799

[Dr-Bean]

Got bitten by this too. Creating a _manylinux.py file works, but imo, a command-line option or even an environment variable to exclude manylinux wheels would be a lot cleaner.

[tlandschoff-scale]

We got bitten by this too. Builds that used to work (on CentOS 5, CentOS 6) still seemed to work but the final PyInstaller build does not work.

[sashkab]

@dstufft -- I'm wondering it you decided on implementing an option to skip manylinux1 wheels download? FOr some reason, I'm getting bitten by this over and over. Keeping _manylinux.py file in site-packages is nice, but I can't remember always add it... :(

[dholth]

There is an issue suggesting that the non-manylinux1 tag should have precedence over the manylinux1 tag. Would that solve your issue?

[asottile]

No it will not. I'd like a flag to ignore manylinux entirely (and download source distribution or other wheels).

[dholth]

Have you tried publishing a package that provides _manylinux.py and installing that into your virtual environments as a matter of course?

[asottile]

Yes, but I consider it a dirty hack (why should installing a package have a side-effect of changing how pip functions?).

It also doesn't work in all cases for example system pip where I do not control dist-packages.

[pfmoore]

If anyone is interested in writing a PR for this, that would probably help move it forward (a command line option would make the most sense, as that would automatically support setting it in the ini file or via an environment variable).

Otherwise, supplying _manylinux.py is probably the best option for now. (You could set PYTHONPATH to a directory of your choosing, and add _manylinux.py there, that would make it visible in all environments)

[asottile]

I started working on a patch, does this seem sane before I start figuring out how to test this?

https://github.com/pypa/pip/compare/master...asottile:no_manylinux?expand=1

For example:

Default

$ pip download libsass --dest foo
Collecting libsass
  Using cached libsass-0.11.1-cp27-cp27mu-manylinux1_x86_64.whl
  Saved ./foo/libsass-0.11.1-cp27-cp27mu-manylinux1_x86_64.whl
Collecting six (from libsass)
  Using cached six-1.10.0-py2.py3-none-any.whl
  Saved ./foo/six-1.10.0-py2.py3-none-any.whl
Successfully downloaded libsass six
$ ls foo/
libsass-0.11.1-cp27-cp27mu-manylinux1_x86_64.whl
six-1.10.0-py2.py3-none-any.whl

With --no-manylinux

$ pip download libsass --dest foo --no-manylinux
Collecting libsass
  Using cached libsass-0.11.1.tar.gz
  Saved ./foo/libsass-0.11.1.tar.gz
Collecting six (from libsass)
  Using cached six-1.10.0-py2.py3-none-any.whl
  Saved ./foo/six-1.10.0-py2.py3-none-any.whl
Successfully downloaded libsass six
$ ls foo/
libsass-0.11.1.tar.gz  six-1.10.0-py2.py3-none-any.whl

[sashkab]

@asottile -- why don't you submit pull request so somebody could review it and provide comments?

[asottile]

Sure, felt like I should get a first round of feedback on the initial approach since it is untested currently but I can do that

[asottile]

https://github.com/pypa/pip/pull/3892

[jayvdb]

Could this be a problem wIth the new manylinux download being run unnecessarily? If you have an existing wheel locally, pip doesnt need to fetch a new wheel. i.e. by default it shouldnt download a manylinux wheel if there is already a non-manylinux wheel available locally.

Then if someone really wants pip to not use a locally available wheel, they shouldnt put it in a place that pip is looking for local wheels.

[sashkab]

Could this be a problem wIth the new manylinux download being run unnecessarily?

This too. But most importantly, I want an option in pip which will disable manylinux completely. Currently, I need to check if manylinux wheel somehow got downloaded and kill it in the wheelhouse. This is very annoying at times.

[asottile]

Our main usecase is we do a one-time download / build of wheels to put in our internal pypi server and manylinux wheels are very much incompatible with our security requirements.

[jayvdb]

Any solution which focuses on manylinux will be linux specific.

manylinux wheels are very much incompatible with our security requirements.

If I understand correctly, you want pip to not download binary from a foreign repo, but you are happy with a binary being built and used locally.

But there could be a Windows shop which has the same security requirements, and a manylinux solution wont work for them.

In which case you want to be able to disable binary for foreign repo, and then the Windows shop will also be able to use the solution.

[asottile]

It's not that it's binary from a foreign repo, it's that shared object files of libraries (that often have security fixes such as libxml, libssl, etc.) are straight vendored into the wheel.

The windows shop is probably already ok with --no-binary ':all:' which'll avoid the win32 / win_amd64 wheels?

[jayvdb]

it's that shared object files of libraries (that often have security fixes such as libxml, libssl, etc.) are straight vendored into the wheel.

Can you give an example?

Maybe there is a generic way to improve pip such that it excludes/rejects those prebuilt wheels, only when they include undesirable contents, which could also occur on Windows.

[sashkab]

@jayvdb example is numpy -- it comes as manylinux wheel, with builtin libraries which aren't compatible with the system and caused me couple hours of headache when I accidentally downloaded manylinux wheel and used it to install numpy. So I'd rather have --no-manylinux flag for pip {download,install,wheel}, rather then I need to waste time later to figure out why something suddenly doesn't work the way it should.

[jayvdb]

https://github.com/numpy/numpy/issues/7570 appears to be the only open issue related to manylinux. It does confirm they are shipping .so's and causing many problems in the process. :/

[xavfernandez]

I'd rather not have a "manylinux" specific option.

Maybe a more general option --only-pure-python or something akin to --no-binary or --only-binary...

[asottile]

Manylinux is already a special case in pip. I also still want to be able to download normal binary wheels (such as from an internal pypi server).

[pfmoore]

Manylinux is already a special case in pip.

Is manylinux not simply a specific compatibility tag? (I'm not that familiar with how manylinux is implemented). I would assume that Linux platforms state that they support a set of compatibility tags that includes manylinux, but that they prefer platform-specific binaries over manylinux. In which case, the more general option would be to have something that allows users to remove tags from the list of supported compatibility tags.

In any case, I agree with @xavfernandez that we should prefer general solutions over special cases. The compatibility tag mechanism handles this (or should, it's what it was designed for) so I'd prefer manylinux to work within that framework (and then this issue becomes "we need a way to override the default platform compatibility list").

[asottile]

Here's where pip special cases manylinux: https://github.com/pypa/pip/blob/a88beb1523f5c5ac9703b7c13296058d3725d54a/pip/pep425tags.py#L278

[pfmoore]

OK. I wonder why it replaces linux with manylinux, rather than just adding a lower-priority manylinux.

[xavfernandez]

@pfmoore it adds a manylinux flavor to the arch in addition to the supported vanilla arch.

We could maybe piggyback on #3760 and allow --platform option for pip install.

[dstufft]

It doesn't replace it. It just adds another manylinux which is preferred over the generic linux. See:

>>> from pip.pep425tags import get_supported
>>> for supported in get_supported():
>>> for supported in get_supported():
...     print(supported)
...
('cp35', 'cp35m', 'manylinux1_x86_64')
('cp35', 'cp35m', 'linux_x86_64')
('cp35', 'abi3', 'manylinux1_x86_64')
('cp35', 'abi3', 'linux_x86_64')
('cp35', 'none', 'manylinux1_x86_64')
('cp35', 'none', 'linux_x86_64')
('py3', 'none', 'manylinux1_x86_64')
('py3', 'none', 'linux_x86_64')
('cp35', 'none', 'any')
('cp3', 'none', 'any')
('py35', 'none', 'any')
('py3', 'none', 'any')
('py34', 'none', 'any')
('py33', 'none', 'any')
('py32', 'none', 'any')
('py31', 'none', 'any')
('py30', 'none', 'any')

If you notice that line of code does two things:

arches = [  # Split over two lines to make it easier to see
    # Add manylinux, which can be computed by swapping the platform tag from the tag
    # returned by distutils
    arch.replace('linux', 'manylinux1'),
    # Add the default platform tag, which is just what distutils returns.
    arch,
]

So yea, manylinux1 isn't special other than we have to implement the generation ourselves rather than relying more on distutils to do it for us. In that vein, I agree that we don't want a manylinux specific option though and we should try to find a more general purpose option.