First attempt at cross-python wheel

[janden]

Following https://github.com/joerick/python-ctypes-package-sample/blob/main/setup.py.

[janden]

@blackwer Still getting different wheels on this one for the same platform. Let me know if you have any ideas.

[lu1and10]

@blackwer Still getting different wheels on this one for the same platform. Let me know if you have any ideas.

just a typo, cmd_class should be cmdclass

[blackwer]

feel free to grab the CI changes from https://github.com/flatironinstitute/finufft/pull/394/files -- they get rid of the other pythons etc.

[janden]

just a typo, cmd_class should be cmdclass

That's embarrassing. Thanks for catching.

[lu1and10]

just a typo, cmd_class should be cmdclass

That's embarrassing. Thanks for catching.

I always make typos, Robert fixed it. Now failing on win, I find https://github.com/mxschmitt/action-tmate useful to debug ci. If the ci fails you can ssh onto the runner and debug by adding following lines to ci yml.

    - name: Setup tmate session
      if: ${{ failure() }}
      uses: mxschmitt/action-tmate@v3

[WardBrian]

I suspect the windows failure is from the CTypesExtension and build_ext changes. These override get_export_symbols to make it so that PyInit_ModuleName is no longer a required export symbol.

This would allow Windows to link the extension without the stub for that function, but it also prevents the module from being imported with importlib which seems to be used to find the path to the shared library.

These aren't really needed for the cross-py wheels, only the bdist_wheel_abi_none function is

[lu1and10]

I suspect the windows failure is from the CTypesExtension and build_ext changes. These override get_export_symbols to make it so that PyInit_ModuleName is no longer a required export symbol.

This would allow Windows to link the extension without the stub for that function, but it also prevents the module from being imported with importlib which seems to be used to find the path to the shared library.

These aren't really needed for the cross-py wheels, only the bdist_wheel_abi_none function is

The importlib can not recognize .so on windows, I tried .pyd for windows, it seems to work.

    def get_ext_filename(self, ext_name):
        if self._ctypes:
            if platform.system() != "Windows":
                return ext_name + ".so"
            else:
                return ext_name + ".pyd"
        return super().get_ext_filename(ext_name)

[WardBrian]

That would work, but I think you can still just remove the CtypesExtension class and extra logic, it is not really buying you anything here. The intention of the class is to unify the file extension and remove the need to make the library importable, but if you're still going to make it importable and customize the file extension back to being platform specific it's not necessary to have

[lu1and10]

That would work, but I think you can still just remove the CtypesExtension class and extra logic, it is not really buying you anything here. The intention of the class is to unify the file extension and remove the need to make the library importable, but if you're still going to make it importable and customize the file extension back to being platform specific it's not necessary to have

Yes, it wouldn't be platform portable. Maybe I miss something, adding the ctypesextension makes just one wheel and one extension binary for all python3.8...3.12 on windows, without the get_ext_filename, it generates finufftc.cp3*-win_amd64.pyd for each python version, with the logic we get one finufftc.pyd and only one wheel for all versions. I guess this PR intends to make python version portable wheel/extension on each platform...

[WardBrian]

The wheels being generic for any Python version is provided by bdist_wheel_abi_none, not the CtypesExtension

[lu1and10]

The wheels being generic for any Python version is provided by bdist_wheel_abi_none, not the CtypesExtension

Yes, the wheel name is by bdist_wheel_abi_none, so we get finufft-2.2.0b0-py3-none-win_amd64.whl, inside the wheel there is a binary extension file finufftc.pyd if with the overided get_ext_filename. Without the overided get_ext_filename, using python 3.9, we get same wheel name finufft-2.2.0b0-py3-none-win_amd64.whl, but the inside binary extension name is finufftc.cp39-win_amd64.pyd, python3.10 can install the wheel, probably will search for finufftc.cp310-win_amd64.pyd and won't recognize finufftc.cp39-win_amd64.pyd, I remember encountered such issue before, may be I'm wrong....

[WardBrian]

Ah, I understand. You could of course avoid having to import the library at all and locate it without importlib, but at that point you might as well avoid using Extension for any part of the build (which I think is something @blackwer wants anyway?)

[lu1and10]

Ah, I understand. You could of course avoid having to import the library at all and locate it without importlib, but at that point you might as well avoid using Extension for any part of the build (which I think is something @blackwer wants anyway?)

Yes, that’s Robert PR #394, shipping the finufft c++ lib as package data so no module extension built, no ctypeextension class needed. If we use PR #394, I guess this PR may not be merged.

[janden]

Superseded by #429.