lefthook icon indicating copy to clipboard operation
lefthook copied to clipboard
verified publisher icon evilmartians

fix: pack one binary per platform into python wheels

Open danfimov opened this issue 5 months ago • 5 comments

Closes #971

Context

Previously, the PyPI package was being built as a single wheel containing binaries for all architectures. This meant every user, regardless of their platform, was downloading a 50+ MB wheel with binaries they would never use. This is inefficient and not the proper way to distribute platform-specific packages on PyPI.

Changes

The fix implements platform-specific wheel generation, where each wheel contains only the binary for its target architecture.

I also replaced license classifier with license_files parameter, because this classifier is deprecated:

/Users/danfimov/Documents/fun/lefthook/packaging/pypi/.venv/lib/python3.12/site-packages/setuptools/config/_apply_pyprojecttoml.py:61: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!

        ********************************************************************************
        Please consider removing the following classifiers in favor of a SPDX license expression:

        License :: OSI Approved :: MIT License

        See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
        ********************************************************************************

!!

I tested building process with this commands:

  • Go to directory with setup.py: cd packaging/pypi
  • Call setup.py to build wheel for my machine: python setup.py bdist_wheel
  • To test with different architecture, I added env vars: LEFTHOOK_TARGET_PLATFORM=linux LEFTHOOK_TARGET_ARCH=arm64 python setup.py bdist_wheel

P.S. I'm familiar with Python, but not with Ruby, so please review part with pack.rb changes carefully)

danfimov avatar Nov 02 '25 13:11 danfimov

Maybe it will be even better to migrate from setup.py completely and use pyproject.toml with build instead:

  • https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
  • https://build.pypa.io/en/stable/

But let's think about it in a different MR after fixing this issue with platform-specific binaries)

danfimov avatar Nov 02 '25 13:11 danfimov

Thank you for preparing this PR! I will review it later this week :+1:

mrexox avatar Nov 05 '25 07:11 mrexox

I have slightly changed the script and built the packages

PYTHON_PLATFORMS = ["linux", "darwin", "freebsd", "openbsd", "windows"].product(["x86_64", "arm64"])

# ...

    PYTHON_PLATFORMS.each do |os, arch|
      puts "Building wheel for #{os}-#{arch}..."

      cd(pypi_dir)
      ENV["LEFTHOOK_TARGET_PLATFORM"] = os
      ENV["LEFTHOOK_TARGET_ARCH"] = arch
      system("python setup.py bdist_wheel", exception: true)
    end

    # cd(pypi_dir)
    # system("python -m twine upload --verbose --repository lefthook dist/*", exception: true)

However when I try to install a package it fails:

pip install packaging/pypi/dist/lefthook-2.0.2-py3-none-darwin_arm64.whl
ERROR: lefthook-2.0.2-py3-none-darwin_arm64.whl is not a supported wheel on this platform.

Also I noticed that logs print info about adding all binaries:

Building wheel for openbsd-arm64...
running bdist_wheel
running build
running build_py
copying lefthook/__init__.py -> build/lib/lefthook
copying lefthook/main.py -> build/lib/lefthook
copying lefthook/__main__.py -> build/lib/lefthook
creating build/lib/lefthook/bin/lefthook-openbsd-arm64
copying lefthook/bin/lefthook-openbsd-arm64/lefthook -> build/lib/lefthook/bin/lefthook-openbsd-arm64
/Users/ian/.local/share/mise/installs/python/3.11.9/lib/python3.11/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
installing to build/bdist.macosx-11.0-arm64/wheel
running install
running install_lib
creating build/bdist.macosx-11.0-arm64/wheel
creating build/bdist.macosx-11.0-arm64/wheel/lefthook
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-openbsd-arm64
copying build/lib/lefthook/bin/lefthook-openbsd-arm64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-openbsd-arm64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-linux-x86_64
copying build/lib/lefthook/bin/lefthook-linux-x86_64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-linux-x86_64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-linux-arm64
copying build/lib/lefthook/bin/lefthook-linux-arm64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-linux-arm64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-freebsd-arm64
copying build/lib/lefthook/bin/lefthook-freebsd-arm64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-freebsd-arm64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-darwin-x86_64
copying build/lib/lefthook/bin/lefthook-darwin-x86_64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-darwin-x86_64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-darwin-arm64
copying build/lib/lefthook/bin/lefthook-darwin-arm64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-darwin-arm64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-openbsd-x86_64
copying build/lib/lefthook/bin/lefthook-openbsd-x86_64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-openbsd-x86_64
creating build/bdist.macosx-11.0-arm64/wheel/lefthook/bin/lefthook-freebsd-x86_64
copying build/lib/lefthook/bin/lefthook-freebsd-x86_64/lefthook -> build/bdist.macosx-11.0-arm64/wheel/./lefthook/bin/lefthook-freebsd-x86_64
copying build/lib/lefthook/__init__.py -> build/bdist.macosx-11.0-arm64/wheel/./lefthook
copying build/lib/lefthook/main.py -> build/bdist.macosx-11.0-arm64/wheel/./lefthook
copying build/lib/lefthook/__main__.py -> build/bdist.macosx-11.0-arm64/wheel/./lefthook
running install_egg_info
running egg_info
writing lefthook.egg-info/PKG-INFO
writing dependency_links to lefthook.egg-info/dependency_links.txt
writing entry points to lefthook.egg-info/entry_points.txt
writing top-level names to lefthook.egg-info/top_level.txt
reading manifest file 'lefthook.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'lefthook.egg-info/SOURCES.txt'
Copying lefthook.egg-info to build/bdist.macosx-11.0-arm64/wheel/./lefthook-2.0.2-py3.11.egg-info
running install_scripts
creating build/bdist.macosx-11.0-arm64/wheel/lefthook-2.0.2.dist-info/WHEEL
creating 'dist/lefthook-2.0.2-py3-none-openbsd_arm64.whl' and adding 'build/bdist.macosx-11.0-arm64/wheel' to it
adding 'lefthook/__init__.py'
adding 'lefthook/__main__.py'
adding 'lefthook/main.py'
adding 'lefthook/bin/lefthook-darwin-arm64/lefthook'
adding 'lefthook/bin/lefthook-darwin-x86_64/lefthook'
adding 'lefthook/bin/lefthook-freebsd-arm64/lefthook'
adding 'lefthook/bin/lefthook-freebsd-x86_64/lefthook'
adding 'lefthook/bin/lefthook-linux-arm64/lefthook'
adding 'lefthook/bin/lefthook-linux-x86_64/lefthook'
adding 'lefthook/bin/lefthook-openbsd-arm64/lefthook'
adding 'lefthook/bin/lefthook-openbsd-x86_64/lefthook'
adding 'lefthook-2.0.2.dist-info/LICENSE'
adding 'lefthook-2.0.2.dist-info/METADATA'
adding 'lefthook-2.0.2.dist-info/WHEEL'
adding 'lefthook-2.0.2.dist-info/entry_points.txt'
adding 'lefthook-2.0.2.dist-info/top_level.txt'
adding 'lefthook-2.0.2.dist-info/RECORD'
removing build/bdist.macosx-11.0-arm64/wheel

Is there anything wrong? I'm not quite familiar with Python build process and will be able to take a look later. But maybe it's clear for you what went wrong 👀

mrexox avatar Nov 13 '25 08:11 mrexox

After some more hours of debug I decided to switch to uv and hatch for package build.

Now it should work. You can test build process like that:

  • Delete or comment string with system("uv publish", exception: true) in pack.rb
  • Call ruby pack.rb prepare and ruby pack.rb publish_pypi
  • There should be separate wheel for every platform + universal wheel

And you can check installation of right wheel from pypi using this package from test pypi: https://test.pypi.org/project/lefthook-test/#files (I will delete it after this MR will be merrged)

There is one problem: I can't build and push to pypi wheels for freebsd / openbsd (only for linux/macos/windows). It happens becase pypi has limited number of supported platform tags (source for list of supported tags). Example of issue:

Uploading lefthook_test-2.0.4-py3-none-freebsd_13_0_aarch64.whl (4.6MiB)
error: Failed to publish `dist/lefthook_test-2.0.4-py3-none-freebsd_13_0_aarch64.whl` to https://test.pypi.org/legacy/
  Caused by: Upload failed with status code 400 Bad Request. Server says: 400 Binary wheel 'lefthook_test-2.0.4-py3-none-freebsd_13_0_aarch64.whl' has an unsupported platform tag 'freebsd_13_0_aarch64'.

So I decided to build platform-specific wheels with one binary for supported platforms + one universal wheel with every binary for other platforms. If someone will try to install lefthook on linux, he will get wheel lefthook-2.0.4-py3-none-manylinux_2_17_x86_64.whl with one binary of lefthook for his platform. If someone will try to install lefthook from pypi on freebsd, he will get universal wheel with binaries for all platforms. You can check this behavior by installing lefthook in some test project:

> uv pip install lefthook-test==2.0.5 --index-url https://test.pypi.org/simple/
Audited 1 package in 0.39ms
> cd .venv/lib/python3.14/site-packages/lefthook/bin/
> ls -la
Permissions Size User     Date Modified Name
drwxrwxr-x     - danfimov 30 Nov 13:27   lefthook-linux-x86_64
.rw-rw-r--     0 danfimov 30 Nov 13:23   .keep

It's not a perfect solution (freebsd user still need to download 50Mb of binaries), but I see only one alternative - write in docs something like "we don't support installation from pypi for platforms other than linux/windows/macos" and just not build universal wheel. And I don't really like to do that. What you think we should do @mrexox?

danfimov avatar Nov 30 '25 12:11 danfimov

I think it's fine to keep 50MB giant for FreeBSD and OpenBSD. I'll try to find a solution if someone complains, but the goal is to deliver lefthook even with lots of odd binaries.

I will check the changes later this week. Thank you for putting an effort into this!

mrexox avatar Dec 01 '25 09:12 mrexox