support pep517 builds - poetry install on Mac fails

[nigelmegitt]

On MacOS 14.3, python 3.11.7, in a project that needs python-snappy as a dependency of another library (in this case, localstack). After installing snappy with brew, the attempt to build python-snappy fails with:

 • Installing python-snappy (0.6.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  /private/var/folders/s1/ls1zsdv57yj_hxx0ldh9h6jh0000gn/T/tmp2shjyr97/.venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'cffi_modules'
    warnings.warn(msg)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-14-x86_64-cpython-311
  creating build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/snappy.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/snappy_cffi.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/__init__.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/hadoop_snappy.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/snappy_formats.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/__main__.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  copying src/snappy/snappy_cffi_builder.py -> build/lib.macosx-14-x86_64-cpython-311/snappy
  running build_ext
  building 'snappy._snappy' extension
  creating build/temp.macosx-14-x86_64-cpython-311
  creating build/temp.macosx-14-x86_64-cpython-311/src
  creating build/temp.macosx-14-x86_64-cpython-311/src/snappy
  clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -I/private/var/folders/s1/ls1zsdv57yj_hxx0ldh9h6jh0000gn/T/tmp2shjyr97/.venv/include -I/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/snappy/crc32c.c -o build/temp.macosx-14-x86_64-cpython-311/src/snappy/crc32c.o
  clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -I/private/var/folders/s1/ls1zsdv57yj_hxx0ldh9h6jh0000gn/T/tmp2shjyr97/.venv/include -I/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/snappy/snappymodule.cc -o build/temp.macosx-14-x86_64-cpython-311/src/snappy/snappymodule.o
  src/snappy/snappymodule.cc:33:10: fatal error: 'snappy-c.h' file not found
  #include <snappy-c.h>
           ^~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  

  at /usr/local/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with python-snappy (0.6.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "python-snappy (==0.6.1)"'.

I can manually cause python-snappy to be built with pip, outside of the pep517 / poetry infrastructure, with a workaround command such as the one in the README, e.g. for me:

CPPFLAGS="-I$(brew --prefix)/include -L$(brew --prefix)/lib" poetry run pip install python-snappy

This works but is really frustrating - I see there are quite a lot of MacOS-related build issues already, and am trying to avoid raising a duplicate.

Obviously I'd prefer that the build script just worked entirely, but even if I had to manually set CPPFLAGS it would be a big improvement if python-snappy were to support PEP 517 builds directly.

[martindurant]

Following #130 , build will no longer be necessary for this library - we just need to get a release out. I was hoping to transfer the repo first (so I can be maintainer), but I suppose there's no real reason to wait.

Also worth pointing out, that conda has always been a more reliable way to get python+binary packages.

[nigelmegitt]

Also worth pointing out, that conda has always been a more reliable way to get python+binary packages.

Are you saying we could work around the issue by not using brew to install snappy, but conda to get and build both snappy and the python-snappy api to it? I was not aware of that as an option.

[nigelmegitt]

Following #130 , build will no longer be necessary for this library - we just need to get a release out.

And for that release to be adopted downstream, of course.

[martindurant]

conda to get and build both snappy and the python-snappy api to it?

Not build, just get :)

And for that release to be adopted downstream

Actually no, unless anyone is pinning versions. It should be a drop-in replacement that looks like a new version of this package, but is calling cramjam.

[nigelmegitt]

unless anyone is pinning versions.

I'm sure a lot of people are. It's a common practice to manage dependencies and allow regression testing etc before updating, to avoid having users get annoyed when their project randomly stops working when nothing apparently changed.

[martindurant]

It's a common practice to manage dependencies and allow regression testing etc before updating

Then they presumably are not using a setup in which build didn't work :)

[nigelmegitt]

Then they presumably are not using a setup in which build didn't work :)

Unfortunately that assumption is not valid. There are differences in build success across platforms.

[gliptak]

please check https://pypi.org/project/python-snappy/0.7.0.dev1/

[joshuataylor]

I can confirm that 0.7.0.dev1 fixes it for me with 3.11.8 and 3.12.2 on MacOS, for ARM64.

Using CPPFLAGS="-I/opt/homebrew/include -L/opt/homebrew/lib" pip install python-snappy works for 0.6.1.