pip installation error

[theonov13]

trafficstars

Hi,

I'm aiming to install mpi4py-fft using pip. However, I ended up with the following error

[...]
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    cimport fftwf_xfftn
            ^
    ------------------------------------------------------------
    
    mpi4py_fft/fftw/fftwf_xfftn.pyx:1:8: 'fftwf_xfftn.pxd' not found

[...]
 Cython.Compiler.Errors.CompileError: /scratch/pip-build-bwiiwole/mpi4py-fft/mpi4py_fft/fftw/fftwf_xfftn.pyx

The error is reproducible under various conditions

• mpi4py_fft versions (master, 2.0.5 - 2.0.0)
• gnu compiler version (11, 9)
• python version (3.6,3.7,3.9,3.10)
• operating systems (Ubuntu 22.04.3 LTS, Red Hat Enterprise Linux release 8.6 (Ootpa))

I would be grateful for any hint on how to resolve this issue.

Best, Sebastian

[theonov13]

Checking out dalcinl/cython3, i.e.,

git checkout dalcinl/cython3
python3 -m pip install -e .

seems to solve this issue. Thus someone might want to merge these changes into main. ;)

[gurcani]

Thanks! This was driving me nuts.

[benlandrum]

There is a PR https://github.com/mpi4py/mpi4py-fft/pull/30 up for this.

[benlandrum]

The author looks pretty busy. @dalcinl, let me know if I can help. Thanks for all your work.

[dalcinl]

@benlandrum Yes, I'm quite busy, my open source workload is becoming unbearable. Are you able to build locally with the changes in #30? Maybe we should think of making a release that solves build issues, and address any issues with our CI infrastructure later.

[benlandrum]

I'm able to make on Python 3.12 after patching the setup.py file according to remove distutils and grab a compiler according to https://github.com/pypa/setuptools/issues/2806#issuecomment-961805789.

The fix is probably version-dependent, so it'd benefit from CI. How bad are the CI problems right now?

[benlandrum]

index d05d89b..983c06c 100644
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,6 @@ import sys
 import re
 import platform
 import sysconfig
-from distutils import ccompiler
 from setuptools import setup
 from setuptools.dist import Distribution
 from setuptools.extension import Extension
@@ -58,9 +57,29 @@ def get_library_dirs():
         append(dirs, prefix, 'lib')
     return dirs
 
+def get_compiler():
+    """
+    Return a new compiler.
+
+    From https://github.com/pypa/setuptools/issues/2806#issuecomment-961805789.
+    """
+    build_ext = Distribution().get_command_obj("build_ext")
+    build_ext.finalize_options()
+
+    # Register an extension to ensure a compiler is created.
+    build_ext.extensions = [Extension("ignored", ["ignored.c"])]
+
+    # Disable building fake extensions.
+    build_ext.build_extensions = lambda: None
+
+    # Populate self.compiler.
+    build_ext.run()
+
+    return build_ext.compiler
+
 def get_fftw_libs():
     """Return FFTW libraries"""
-    compiler = ccompiler.new_compiler()
+    compiler = get_compiler()
     library_dirs = get_library_dirs()
     libs = {}
     for d in ('float', 'double', 'long double'):

[benlandrum]

Enabled the pipelines on a fork of mine https://github.com/benlandrum/mpi4py-fft. Will give them a shot and get back to you. Running into numpy module problems, likely familiar to you.

[benlandrum]

My patch above was probably in the wrong direction. Please ignore.

I started from your branch and edited the FFTW setup. It now runs on all Python versions, all MPI implementations, and on Ubuntu and macOS.

Please let me know what you think, @dalcinl. https://github.com/mpi4py/mpi4py-fft/pull/40