A new release for soundfile

[bastibe]

This is a continuation of the discussion in #310, where I'm working on packaging a new version of soundfile.

It appears that the packaging problem is more complicated than anticipated. Our existing build system will probably need to be replaced, which will take some time.

If anyone would like to help me package soundfile, I'd be grateful!

The problem is, we need to package four wheels for

1. Windows 32 Bit
2. Windows 64 Bit
3. macOS Intel
4. macOS Arm

each containing their own libsndfile{.dll,.dylib}, and making sure that the right version gets installed on the right platform.

We have a build script in place for building packages for 1-3, but I don't yet know how to build wheels specifically for macOS Arm, and it seems that we should convert our build system to use PEP 517 (pyproject.toml/build et al).

I only had an hour or so today to work on this, so there wasn't much progress. I'll keep you updated.

[chuma9615]

Thanks for taking time to solve this, If u need any helping hands let me know

[HexDecimal]

pyproject.toml is super nice but I don't think it will resolve your current issues with wheels. What it can do is make it so that cffi is installed before the setup.py script is run which would crash otherwise.

I'm not sure about the specific CLI flags. I think it's adding -arch arm64 to the C/link flags will work it if the tools support it. Or -arch arm64 -arch x86_64 for a Universal 2 library.

The platform tags would be something like macosx_11_0_arm64. If the library supports both arm64 and x86_64 then the platform tag would be macosx_10_9_universal2 instead (pulling examples from my own projects, I don't know how to figure out the best version numbers.)

For my own projects I have to rely on GitHub Actions to build MacOS packages/libraries for me as downloadable artifacts. I can write a workflow for you if you're unable to compile the arm64 library yourself.

[bastibe]

Thank you so much for your help!

If you could point me to an example of how to have Github compile the macOS libraries, that would be fantastic! That might actually allow us to compile the wheels separately on each OS, instead of the current franken-system that packages precompiled libraries with the generic python code.

[HexDecimal]

This would be an example of a workflow. This script would be stored in a path like .github/workflows/build-libs.yml in the repository, and will take affect as soon as it's pushed to the repo.

name: Compile libraries

on:
  push:
  pull_request:

jobs:
  build-libs:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: ["macos-11"]
      fail-fast: true

    steps:
      - uses: actions/checkout@v2
      - name: Compile library
        run: ./mac_build.sh
      - uses: actions/upload-artifact@v2
        with:
          name: libs
          path: "*.dylib"
          retention-days: 7
          if-no-files-found: error 

I imagine this running the mac_build.sh script from bastibe/libsndfile-binaries, so this workflow would be added there for this example. You can also see the docs for actions/upload-artifact.

GitHub has good documentation on making workflows. Build artifacts uploaded this way can be downloaded from another job, such as a job that builds the wheel.

[bastibe]

This worked amazingly well, thank you a whole lot!

At the moment, Github runners do not yet provide M1 machines, but that's not super important at the moment, since @chuma9615 and @jurihock have already gracefully provided pre-build libraries in libsndfile-binaries#7 and libsndfile-binaries#8, respectively.

I'll investigate cross-compilation once I've managed to build wheels.

[bastibe]

I have drafted a new release at releases/0.11.0b1.

There is a known issue that will require a recompile of the Windows binaries with Visual Studio 14+, and will probably never work for all versions of Python. Apparently, file descriptor open does only work if libsndfile is compiled against the same Microsoft C Runtime as Python. The present libsndfile-provided binaries are apparently incompatible with my Python.

If anyone would like to contribute up-to-date Python libraries compiled with Visual Studio 14+, I'd be grateful for a pull request in bastibe/libsndfile-libraries. Or better yet, integrate it into the CI scripts to build the binaries automatically.

Also, you would help me tremendously if anyone could test the provided wheels on macOS and M1!

[SpotlightKid]

Will you provide source distribution archives (tarballs) for the latest and upcoming releases or should Linux distro packagers use the source archives automatically created by Github?

[bastibe]

I will upload a source distribution.

The reason I didn't attach it to the draft release linked above is that the current build script still bundles all the libraries with the source dist, making it unnecessarily big. This will need fixing before the final release is cut.

[HexDecimal]

There is a known issue that will require a recompile of the Windows binaries with Visual Studio 14+, and will probably never work for all versions of Python. Apparently, file descriptor open does only work if libsndfile is compiled against the same Microsoft C Runtime as Python. The present libsndfile-provided binaries are apparently incompatible with my Python.

Looks like if you compile to v140 then that binary will support all versions of Python after 3.5. The instructions I've found for telling MinGW to use specific runtimes seems convoluted so maybe it would be easier to use Vcpkg to get these builds.

[HaHeho]

Or better yet, integrate it into the CI scripts to build the binaries automatically.

I'm inexperienced with all the required compilation steps and CI. But I remembered that Mathias uses a script which builds a small version of libsndfile on Mac during some Github actions, build-small-libsndfile.sh. Forgive me if this doesn't help at all. :)

[bastibe]

@SpotlightKid I just uploaded a fixed sdist to the latest tag.

Right now, all that's missing for a release is

• someone should test the M1 wheels.
• I just noticed that the macos x64 wheel does not install. Any help as to why would be greatly appreciated.
• I've opened a pull request for the new release in #326 and noticed that the automated tests no longer run. Does anyone know why?

@HaHeho thank you for your response. Thanks to @HexDecimal, we now have automated builds for Windows and macOS! That's wild!

[HexDecimal]

... and noticed that the automated tests no longer run. Does anyone know why?

If you check the commits on GitHub then TravisCI's last status check was Nov 26, 2019. It stopped working sometime after that, probably when they dropped support for the travis-ci.org site. You'd have to migrate to travis-ci.com, but I don't think that site actually gives free time for FOSS projects. You might get a one time allotment to work with while you move to something else. I remember dropping TravisCI for GitHub Actions over this.

More info: https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567

[SpotlightKid]

@SpotlightKid I just uploaded a fixed sdist to the latest tag.

Thanks for the heads up. The URLs for the attached files still have 0.11.0b1 in the path, which is slightly inconvenient. But I'm assuming this will be correct when a proper release is made.

[chuma9615]

@bastibe Tested the wheel on a M1 machine and it's working

Note that I tested soundfile-0.11.0b2-py2.py3-none-any.whl

[bastibe]

@chuma9615 that's the source-only wheel. Could you try the M1-specific soundfile-0.11.0b2-py2.py3-none-macosx-10.x-arm64.whl?

[HexDecimal]

Could you try the M1-specific soundfile-0.11.0b2-py2.py3-none-macosx-10.x-arm64.whl?

Note that macosx-10.x-arm64 isn't a valid platform tag since the dash - is used to separate tags from each other and this should all be one platform tag. The x and . in 10.x also isn't valid. Pip and PyPI might panic when seeing the tags in this state.

The libs were built with MACOSX_DEPLOYMENT_TARGET=10.9 so maybe you meant this tag to be macosx_10_9_arm64 and so the wheel would be called soundfile-0.11.0b2-py2.py3-none-macosx_10_9_arm64.whl

How wheel tags work is clear, but MacOS specific info is harder to find: https://www.python.org/dev/peps/pep-0425/ https://github.com/MacPython/wiki/wiki/Spinning-wheels https://lepture.com/en/2014/python-on-a-hard-wheel https://snarky.ca/the-challenges-in-designing-a-library-for-pep-425/

[chuma9615]

@bastibe When using the macOS ARM wheel it crashes

ERROR: soundfile-0.11.0b2-py2.py3-none-macosx-10.x-arm64.whl is not a supported wheel on this platform.

[msuch]

@bastibe there was a tiny typo bug in the code when importing sndfile on Mac. I fixed this in #328.

[azuwis]

I was trying to package python-soundfile 1.11.0 (https://github.com/bastibe/python-soundfile/tree/bastibe/0.11.0) for Nix on macOS arm64, and encountered the not a supported wheel problem:

Full log

Sourcing python-remove-tests-dir-hook
Sourcing python-catch-conflicts-hook.sh
Sourcing python-remove-bin-bytecode-hook.sh
Sourcing setuptools-build-hook
Using setuptoolsBuildPhase
Using setuptoolsShellHook
Sourcing pip-install-hook
Using pipInstallPhase
Sourcing python-imports-check-hook.sh
Using pythonImportsCheckPhase
Sourcing python-namespaces-hook
Sourcing setuptools-check-hook
Using setuptoolsCheckPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/vrckfw0c4wg41ak9nik5gymaigx6i11j-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/test_soundfile.py
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Executing setuptoolsBuildPhase
running bdist_wheel
running build
running build_py
file _soundfile.py (for module _soundfile) not found
creating build
creating build/lib
copying soundfile.py -> build/lib
package init file '_soundfile_data/__init__.py' not found (or not a regular file)
file _soundfile.py (for module _soundfile) not found
generating cffi module 'build/lib/_soundfile.py'
installing to build/bdist.macosx-10.6-arm64/wheel
running install
running install_lib
creating build/bdist.macosx-10.6-arm64
creating build/bdist.macosx-10.6-arm64/wheel
copying build/lib/soundfile.py -> build/bdist.macosx-10.6-arm64/wheel
copying build/lib/_soundfile.py -> build/bdist.macosx-10.6-arm64/wheel
running install_egg_info
running egg_info
creating soundfile.egg-info
writing soundfile.egg-info/PKG-INFO
writing dependency_links to soundfile.egg-info/dependency_links.txt
writing requirements to soundfile.egg-info/requires.txt
writing top-level names to soundfile.egg-info/top_level.txt
writing manifest file 'soundfile.egg-info/SOURCES.txt'
reading manifest file 'soundfile.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'soundfile.egg-info/SOURCES.txt'
Copying soundfile.egg-info to build/bdist.macosx-10.6-arm64/wheel/soundfile-0.11.0-py3.9.egg-info
running install_scripts
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build/bdist.macosx-10.6-arm64/wheel/soundfile-0.11.0.dist-info/WHEEL
creating 'dist/soundfile-0.11.0-py2.py3-none-macosx-10.x-arm64.whl' and adding 'build/bdist.macosx-10.6-arm64/wheel' to it
adding '_soundfile.py'
adding 'soundfile.py'
adding 'soundfile-0.11.0.dist-info/LICENSE'
adding 'soundfile-0.11.0.dist-info/METADATA'
adding 'soundfile-0.11.0.dist-info/WHEEL'
adding 'soundfile-0.11.0.dist-info/top_level.txt'
adding 'soundfile-0.11.0.dist-info/RECORD'
removing build/bdist.macosx-10.6-arm64/wheel
Finished executing setuptoolsBuildPhase
@nix { "action": "setPhase", "phase": "installPhase" }
installing
Executing pipInstallPhase
/private/tmp/nix-build-python3.9-soundfile-0.11.0.drv-0/source/dist /private/tmp/nix-build-python3.9-soundfile-0.11.0.drv-0/source
ERROR: soundfile-0.11.0-py2.py3-none-macosx-10.x-arm64.whl is not a supported wheel on this platform.

After replace macosx-10.x-arm64 in setup.py with any or macosx_11_0_arm64, it built fine, but pytest failed with many MemoryError:

__________________ ERROR at setup of test_blocks_rplus[obj] ___________________

file_stereo_rplus = <_io.FileIO name='tests/delme.please' mode='rb+' closefd=True>

    @pytest.fixture
    def sf_stereo_rplus(file_stereo_rplus):
>       with sf.SoundFile(file_stereo_rplus, 'r+') as f:

tests/test_soundfile.py:124: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
soundfile.py:646: in __init__
    self._file = self._open(file, mode_int, closefd)
soundfile.py:1197: in _open
    file_ptr = _snd.sf_open_virtual(self._init_virtual_io(file),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = SoundFile(<_io.FileIO name='tests/delme.please' mode='rb+' closefd=True>, mode='r+', samplerate=0, channels=0, format='FILE', subtype='FILE', endian='FILE')
file = <_io.FileIO name='tests/delme.please' mode='rb+' closefd=True>

    def _init_virtual_io(self, file):
        """Initialize callback functions for sf_open_virtual()."""
        @_ffi.callback("sf_vio_get_filelen")
>       def vio_get_filelen(user_data):
E       MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks

soundfile.py:1217: MemoryError

According to https://cffi.readthedocs.io/en/latest/using.html#callbacks, ffi.callback() is old style callback, and has many drawbacks, should be replaced by new style callback def_extern().

[bastibe]

Thank you all so much for your help with this release!

And especially thank you @HexDecimal, for implementing the CI builds and tests!

I've uploaded a new set of pre-release wheels: 0.11.0b3.

I hope that these wheels now work on macOS M1. Again, I'd be grateful if anyone with access to an M1 Mac could test the macOS ARM wheel.

[azuwis]

Thank you all so much for your help with this release!

And especially thank you @HexDecimal, for implementing the CI builds and tests!

I've uploaded a new set of pre-release wheels: 0.11.0b3.

I hope that these wheels now work on macOS M1. Again, I'd be grateful if anyone with access to an M1 Mac could test the macOS ARM wheel.

$ pip install https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl is not a supported wheel on this platform.

MacBook Pro (13-inch, M1, 2020), macOS 12.2.1

[bastibe]

According to https://cffi.readthedocs.io/en/latest/using.html#callbacks, ffi.callback() is old style callback, and has many drawbacks, should be replaced by new style callback def_extern().

I've looked into this, and actually drafted an implementation of the callbacks locally. However, there is a problem with this approach: Currently, we are using the ABI mode of CFFI. The "new style" callbacks you mentioned require that soundfile uses the API mode of CFFI, which means compiling a bespoke version of soundfile for every operating system and version of Python we support.

While I am certainly open to doing this, especially now that we have CI runners to automate the task, I think it is out-of-scope for this pull request. Please raise the issue in a new issue, or preferably, try to draft a pull request for it.

[bastibe]

$ pip install https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl is not a supported wheel on this platform.

MacBook Pro (13-inch, M1, 2020), macOS 12.2.1

I've seen a similar error on my wife's old mac. But it seemed to be related to her old version of pip. Perhaps that was wrong. Does your error go away if you install a current version of pip?

Regardless, the issue still needs fixing. I set the OS tag to macosx_10_9_XXX since it seemed to work on my wife's mac, and seemed preferable to macosx_10_9_XXX.macosx_10_10_XXX.macosx_11_0_XXX.future_macosx_that_we_dont_even_know_yet_XXX. But if that's what it takes then I'll do that instead. How annoying.

Could you confirm the correct OS tag to use? Perferably with some measure of backwards and forwards compatibility.

Anyway, my time to work on this for this week is up, so I'll have to defer the issue for next wednesday.

[azuwis]

I've seen a similar error on my wife's old mac. But it seemed to be related to her old version of pip. Perhaps that was wrong. Does your error go away if you install a current version of pip?

No.

bash-3.2$ pip install -U pip
Requirement already satisfied: pip in ./lib/python3.8/site-packages (22.0.3)
bash-3.2$ curl -sLO https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
bash-3.2$ pip install soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl
ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl is not a supported wheel on this platform.

Could you confirm the correct OS tag to use? Perferably with some measure of backwards and forwards compatibility.

macosx_11_0_arm64 works for arm64.

bash-3.2$ mv soundfile-0.11.0b3-py2.py3-none-macosx_10_9_arm64.whl soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
bash-3.2$ pip install soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
Processing ./soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
Requirement already satisfied: cffi>=1.0 in ./lib/python3.8/site-packages (from soundfile==0.11.0b3) (1.15.0)
Requirement already satisfied: pycparser in ./lib/python3.8/site-packages (from cffi>=1.0->soundfile==0.11.0b3) (2.21)
Installing collected packages: soundfile
Successfully installed soundfile-0.11.0b3

[bastibe]

I changed the platform tags to

• Intel: macosx_10_9_x86_64.macosx_11_0_x86_64
• M1: macosx_11_0_arm64

Please check them out again in the 0.11.b3 beta release. If I remember correctly, there was no M1 before macOS 11.0, so we don't need to include the 10.9 tag, right?

[azuwis]

Please check them out again in the 0.11.b3 beta release. If I remember correctly, there was no M1 before macOS 11.0, so we don't need to include the 10.9 tag, right?

It works after upgrading pip:

bash-3.2$ curl -sLO https://github.com/bastibe/python-soundfile/releases/download/0.11.0b3/soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
bash-3.2$ pip3 install soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl 
ERROR: soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl is not a supported wheel on this platform.
WARNING: You are using pip version 20.2.3; however, version 22.0.3 is available.
You should consider upgrading via the '/private/tmp/venv/bin/python3 -m pip install --upgrade pip' command.
bash-3.2$ pip3 install -U pip
Collecting pip
  Using cached pip-22.0.3-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-22.0.3
bash-3.2$ pip3 install soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl 
Processing ./soundfile-0.11.0b3-py2.py3-none-macosx_11_0_arm64.whl
Collecting cffi>=1.0
  Using cached cffi-1.15.0.tar.gz (484 kB)
  Preparing metadata (setup.py) ... done
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Using legacy 'setup.py install' for cffi, since package 'wheel' is not installed.
Installing collected packages: pycparser, cffi, soundfile
  Running setup.py install for cffi ... done
Successfully installed cffi-1.15.0 pycparser-2.21 soundfile-0.11.0b3

[bastibe]

Thank you for the confirmation!

Does anyone know why we need an up-to-date pip for this?

[HexDecimal]

Does anyone know why we need an up-to-date pip for this?

How tags are handed can change from version to version. My guess is that older versions of Pip are looking for exact MacOS versions rather than compatible versions or something similar. If that's the case then adding redundant versions would solve the issue for older versions of pip.

The links I've posted previously should help: https://github.com/bastibe/python-soundfile/issues/325#issuecomment-1044392935, mostly with which MacOS versions Python might be expecting, which could be 10.6, 10.7, or 10.9.

I think adding tags for 10.9 in addition to 11.0 might help. For example the platform tag macosx_10_9_arm64.macosx_11_0_arm64 instead of macosx_11_0_arm64.

[bastibe]

I will try that. Thank you.

[bastibe]

libsndfile 1.1.0 just released, so now it's time to rebuild all the libraries with MP3 support, and then cut the release.