Binary Wheels not linking against libpython

[agronholm]

Originally reported by: Armin Ronacher (Bitbucket: mitsuhiko, GitHub: mitsuhiko)

---

I'm currently using various different hacks that sometimes fail to create wheels with the following qualities:

• they are manylinux1 and mac wheels
• they use cffi to load shared libraries
• they do not link against libpython

Sadly they cannot be generated currently. The only way that sort of works is to put the shared libraries into the purelib root and it's not really purelib, is it.

Ideally there was a way to say: this wheel is py2.py3, has a manylinux/mac platform but no python abi.

---

• Bitbucket: https://bitbucket.org/pypa/wheel/issue/185

[agronholm]

Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth):

---

I was looking at the get_tag() function. It appears you can already specify --plat-name= while building a 'pure' wheel if you want. This is of course less convenient than having wheel find the current platform name automatically, and is 'less pure' than using platlib.

[agronholm]

Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth):

---

It would probably look a bit like https://bitbucket.org/pypa/wheel/pull-requests/69/add-py-limited-api-flag-to-control-abi3/diff

I'm surprised we haven't done this already. Seems like it has come up in the past. I'm personally a big fan of cffi.

[agronholm]

@mitsuhiko Could you point me to a project I could use to verify that the issue is still relevant?

[agronholm]

Can we automatically somehow detect cases such as this?

[mitsuhiko]

@agronholm you can look at getsentry/symbolic which runs into this issue. We solved it by patching the hell out of bdist_wheel through our getsentry/milksnake library.

[agronholm]

Ok so let me just ask: what would you like me to change in wheel so I could close this issue?

[mitsuhiko]

@agronholm it needs a mode where you can define that your wheel contains binary data but is not linked against libpython. Effectively have something that lets me do what this does: https://github.com/getsentry/milksnake/blob/8351a8aa4ac800c586733602a55c1c02f0d19923/milksnake/setuptools_ext.py#L261-L278

[agronholm]

Hm, how about a universal_binary option (can anyone think of a better name?)? All it would do is add the architecture tag to the wheel. It would only take effect if no (traditional) C extensions are present.

I also thought about automatic detection of this condition but it would be too error prone.

[proppy]

@agronholm would that also work with app that bundle .so and load them with ctypes?

Currently, I'm bundle the .so as package_data and relying on @mitsuhiko trick to flag the wheel as not-pure.

But it doesn't trick auditwheel tool and it will continue to raise WheelToolsError('Cannot add platforms to pure wheel')

[mitsuhiko]

@proppy sadly there are limitations in distutils itself which means this flag can only be flagged by building a c extension. So what milksnake does not is building an empty C file and then later overrides the .so with the one from the milksnake build process.

[proppy]

@mitsuhiko is that something that could be patched up in wheel or does it have to be fixed in distutils?

[proppy]

@agronholm @mitsuhiko after looking at distutils source code and stealing a lot of milksnake awesome magic, I ended up the following workaround:

class UniversalBinaryDistribution(Distribution):
    def __init__(self, *args, **kwargs):
        super(UniversalBinaryDistribution, self).__init__(*args, **kwargs)
        base_bdist_wheel = self.cmdclass.get('bdist_wheel', bdist_wheel)
        class UniversalBdistWheel(base_bdist_wheel):
          def get_tag(self):
              rv = base_bdist_wheel.get_tag(self)
              return ('py2.py3', 'none',) + rv[2:]
        self.cmdclass['bdist_wheel'] = UniversalBdistWheel
        base_build_ext = self.cmdclass.get('build_ext', build_ext)
        class UniversalBuildExt(base_build_ext):
            def build_extension(self, ext):
                ext_filename = self.get_ext_filename(ext.name)
                shutil.copy2(ext_filename,
                             os.path.join(self.build_lib, ext_filename))
            def get_ext_filename(self, ext_name):
                ext_path = ext_name.split('.')
                ext_suffix = new_compiler().shared_lib_extension
                return os.path.join(*ext_path) + ext_suffix
        self.cmdclass['build_ext'] = UniversalBuildExt


setup(...
    ext_modules=[
        Extension('module.path.to..precompiledlib.without.so',
                  sources = [])
    ],
    distclass=UniversalBinaryDistribution,
)

It replaces the C compilation process with a shutil.copy2 and the resulting wheels seems to pass auditwheel validation.

Is that something we could consolidate somewhere? (either in distutils or wheels)

[mitsuhiko]

One problem is that a lot of code just checks for .extension_modules or what that attribute is called. If it's empty it assumes pure eggs/wheels. There is a ton of code that just does that. All the workarounds I found break in fancy ways sadly. Even if auditwheel passes there is still embedded metadata which is kinda wrong.

[agronholm]

What metadata does that affect? IIRC wheel itself has the final say over the resulting wheel filename, so that's something I can fix here.

[proppy]

@agronholm the metadata that changed for me was: Root-Is-Purelib: false in /WHEEL

@mitsuhiko are you referring to the ext_modules parameter for the setup() call or build_ext.extensions internal variable?

[mitsuhiko]

The extensions parameter that is set on build_ext and others I think. I would need to look into it again though.