mtuq icon indicating copy to clipboard operation
mtuq copied to clipboard
verified publisher icon uafgeotools

Bug cpython c_ext_L2.c compilation

Open SeismoFelix opened this issue 9 months ago • 8 comments

Hello,

Something happened in the latest approved pull request that changed the compilation of c_ext_L2.c during the execution of

conda env update --file env.yaml

After the installation, the file c_ext_L2.cpython-312-darwin.so created in the same directory where c_ext_L2.c is located, did not work. I got this error message when I simply tried to import a mtuq library.

from mtuq import read /Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/init.py:13: SyntaxWarning: invalid escape sequence '|' """ Waveform misfit function Traceback (most recent call last): File "", line 1, in File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/init.py", line 35, in from mtuq.misfit import PolarityMisfit, WaveformMisfit File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/init.py", line 2, in from mtuq.misfit.waveform import WaveformMisfit File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/init.py", line 5, in from mtuq.misfit.waveform import level0, level1, level2 File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/level2.py", line 14, in from mtuq.misfit.waveform import c_ext_L2 ImportError: dlopen(/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so, 0x0002): tried: '/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so' (malformed load command #14 of 16 at 0x10258f610 with mh=0x10258f000, size (0x0) too small), '/System/Volumes/Preboot/Cryptexes/OS/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so' (no such file), '/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so' (malformed load command #14 of 16 at 0x10258f610 with mh=0x10258f000, size (0x0) too small)

Fortunately, I had installed mtuq before the latest commit and I changed the c_ext_L2.cpython-312-darwin.so file that was not working for another one compiled in a different mtuq environment from a previous installation. Then eveything worked as expected (at least with the examples).

Of course, this is a temporary patch that only works for me, but I have not found what may have changed that caused this behavior.

I tested the installation in two MacOS computers and the error is the same.

Thanks for your help.

Image

Image

SeismoFelix avatar Apr 05 '25 02:04 SeismoFelix

Hi Felix, What is the result of running build_ext.sh? Ryan

On Fri, Apr 4, 2025 at 8:39 PM Felix Rodriguez Cardozo < @.***> wrote:

Hello,

Something happened in the latest approved pull request that changed the compilation of c_ext_L2.c https://github.com/mtuqorg/mtuq/blob/master/mtuq/misfit/waveform/c_ext_L2.cduring the execution of

conda env update --file env.yaml

After the installation, the file c_ext_L2.cpython-312-darwin.so created in the same directory where c_ext_L2.c is located, did not work. I got this error message when I simply tried to import a mtuq library.

from mtuq import read /Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/init.py:13: SyntaxWarning: invalid escape sequence '|' """ Waveform misfit function Traceback (most recent call last): File "", line 1, in File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/init.py", line 35, in from mtuq.misfit import PolarityMisfit, WaveformMisfit File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/init.py", line 2, in from mtuq.misfit.waveform import WaveformMisfit File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ init.py", line 5, in from mtuq.misfit.waveform import level0, level1, level2 File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/level2.py", line 14, in from mtuq.misfit.waveform import c_ext_L2 ImportError: dlopen(/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so, 0x0002): tried: '/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so' (malformed load command #14 https://github.com/mtuqorg/mtuq/pull/14 of 16 at 0x10258f610 with mh=0x10258f000, size (0x0) too small), '/System/Volumes/Preboot/Cryptexes/OS/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so' (no such file), '/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so' (malformed load command #14 https://github.com/mtuqorg/mtuq/pull/14 of 16 at 0x10258f610 with mh=0x10258f000, size (0x0) too small)

Fortunately, I had installed mtuq before the latest commit and I changed the c_ext_L2.cpython-312-darwin.so file that was not working for another one compiled in a different mtuq environment from a previous installation. Then eveything worked as expected (at least with the examples).

Of course, this is a temporary patch that only works for me, but I have not found what may have changed that caused this behavior.

I tested the installation in two MacOS computers and the error is the same.

Thanks for your help. Screenshot.2025-04-04.at.10.35.13.PM.png (view on web) https://github.com/user-attachments/assets/a0552912-6203-4cb2-ac51-7316f8fd6b90

Screenshot.2025-04-04.at.10.35.01.PM.png (view on web) https://github.com/user-attachments/assets/116c4f3f-b576-406e-b34e-3beece684f3a

— Reply to this email directly, view it on GitHub https://github.com/mtuqorg/mtuq/issues/307, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCGSSXORJKMTWP5H6M7CC32X466JAVCNFSM6AAAAAB2P7QQLOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE3TGNZXGE3DCNY . You are receiving this because you are subscribed to this thread.Message ID: @.***> [image: SeismoFelix]SeismoFelix created an issue (mtuqorg/mtuq#307) https://github.com/mtuqorg/mtuq/issues/307

Hello,

Something happened in the latest approved pull request that changed the compilation of c_ext_L2.c https://github.com/mtuqorg/mtuq/blob/master/mtuq/misfit/waveform/c_ext_L2.cduring the execution of

conda env update --file env.yaml

After the installation, the file c_ext_L2.cpython-312-darwin.so created in the same directory where c_ext_L2.c is located, did not work. I got this error message when I simply tried to import a mtuq library.

from mtuq import read /Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/init.py:13: SyntaxWarning: invalid escape sequence '|' """ Waveform misfit function Traceback (most recent call last): File "", line 1, in File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/init.py", line 35, in from mtuq.misfit import PolarityMisfit, WaveformMisfit File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/init.py", line 2, in from mtuq.misfit.waveform import WaveformMisfit File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ init.py", line 5, in from mtuq.misfit.waveform import level0, level1, level2 File "/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/level2.py", line 14, in from mtuq.misfit.waveform import c_ext_L2 ImportError: dlopen(/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so, 0x0002): tried: '/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so' (malformed load command #14 https://github.com/mtuqorg/mtuq/pull/14 of 16 at 0x10258f610 with mh=0x10258f000, size (0x0) too small), '/System/Volumes/Preboot/Cryptexes/OS/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so' (no such file), '/Users/felix/Documents/Software/mtuq_test/mtuq/mtuq/misfit/waveform/ c_ext_L2.cpython-312-darwin.so' (malformed load command #14 https://github.com/mtuqorg/mtuq/pull/14 of 16 at 0x10258f610 with mh=0x10258f000, size (0x0) too small)

Fortunately, I had installed mtuq before the latest commit and I changed the c_ext_L2.cpython-312-darwin.so file that was not working for another one compiled in a different mtuq environment from a previous installation. Then eveything worked as expected (at least with the examples).

Of course, this is a temporary patch that only works for me, but I have not found what may have changed that caused this behavior.

I tested the installation in two MacOS computers and the error is the same.

Thanks for your help. Screenshot.2025-04-04.at.10.35.13.PM.png (view on web) https://github.com/user-attachments/assets/a0552912-6203-4cb2-ac51-7316f8fd6b90

Screenshot.2025-04-04.at.10.35.01.PM.png (view on web) https://github.com/user-attachments/assets/116c4f3f-b576-406e-b34e-3beece684f3a

— Reply to this email directly, view it on GitHub https://github.com/mtuqorg/mtuq/issues/307, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCGSSXORJKMTWP5H6M7CC32X466JAVCNFSM6AAAAAB2P7QQLOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE3TGNZXGE3DCNY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

rmodrak avatar Apr 05 '25 14:04 rmodrak

Thanks Ryan,

Fun thing: It doe not work because the instruction: rm mtuq/misfit/*.so does not work because the *so file is in mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so. But it has been like this always.

I modified the file to be rm mtuq/misfit/waveform/*.so and I got this:

/Users/felix/Documents/software/mtuq_felix/mtuq/./setup.py:7: SetuptoolsDeprecationWarning: The test command is disabled and references to it are deprecated. !!

    ********************************************************************************
    Please remove any references to `setuptools.command.test` in all supported versions of the affected package.

    This deprecation is overdue, please update your project and remove deprecated
    calls to avoid build errors in the future.
    ********************************************************************************

!! from setuptools.command.test import test as test_command running build_ext building 'mtuq.misfit.waveform.c_ext_L2' extension creating build/temp.macosx-10.13-x86_64-cpython-312/mtuq/misfit/waveform x86_64-apple-darwin13.4.0-clang -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /Users/felix/anaconda3/envs/mtuq_felix/include -D_FORTIFY_SOURCE=2 -isystem /Users/felix/anaconda3/envs/mtuq_felix/include -I/Users/felix/anaconda3/envs/mtuq_felix/lib/python3.12/site-packages/numpy/core/include -I/Users/felix/anaconda3/envs/mtuq_felix/include/python3.12 -c mtuq/misfit/waveform/c_ext_L2.c -o build/temp.macosx-10.13-x86_64-cpython-312/mtuq/misfit/waveform/c_ext_L2.o -Ofast In file included from mtuq/misfit/waveform/c_ext_L2.c:3: In file included from /Users/felix/anaconda3/envs/mtuq_felix/lib/python3.12/site-packages/numpy/core/include/numpy/arrayobject.h:5: In file included from /Users/felix/anaconda3/envs/mtuq_felix/lib/python3.12/site-packages/numpy/core/include/numpy/ndarrayobject.h:12: In file included from /Users/felix/anaconda3/envs/mtuq_felix/lib/python3.12/site-packages/numpy/core/include/numpy/ndarraytypes.h:1929: /Users/felix/anaconda3/envs/mtuq_felix/lib/python3.12/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings] 17 | #warning "Using deprecated NumPy API, disable it with "
| ^ mtuq/misfit/waveform/c_ext_L2.c:144:17: warning: use of infinity via a macro is undefined behavior due to the currently enabled floating-point options [-Wnan-infinity-disabled] 144 | next_iter = INFINITY; | ^ 2 warnings generated. creating build/lib.macosx-10.13-x86_64-cpython-312/mtuq/misfit/waveform x86_64-apple-darwin13.4.0-clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/felix/anaconda3/envs/mtuq_felix/lib -L/Users/felix/anaconda3/envs/mtuq_felix/lib -Wl,-rpath,/Users/felix/anaconda3/envs/mtuq_felix/lib -L/Users/felix/anaconda3/envs/mtuq_felix/lib -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/felix/anaconda3/envs/mtuq_felix/lib -L/Users/felix/anaconda3/envs/mtuq_felix/lib -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -isystem /Users/felix/anaconda3/envs/mtuq_felix/include -D_FORTIFY_SOURCE=2 -isystem /Users/felix/anaconda3/envs/mtuq_felix/include build/temp.macosx-10.13-x86_64-cpython-312/mtuq/misfit/waveform/c_ext_L2.o -o build/lib.macosx-10.13-x86_64-cpython-312/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so ld: warning: duplicate -rpath '/Users/felix/anaconda3/envs/mtuq_felix/lib' ignored ld: warning: duplicate -rpath '/Users/felix/anaconda3/envs/mtuq_felix/lib' ignored copying build/lib.macosx-10.13-x86_64-cpython-312/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so -> mtuq/misfit/waveform

Unfortunately, I got the same error when I try to import mtuq read

from mtuq import read Traceback (most recent call last): File "", line 1, in File "/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/init.py", line 35, in from mtuq.misfit import PolarityMisfit, WaveformMisfit File "/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/init.py", line 2, in from mtuq.misfit.waveform import WaveformMisfit File "/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/waveform/init.py", line 5, in from mtuq.misfit.waveform import level0, level1, level2 File "/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/waveform/level2.py", line 14, in from mtuq.misfit.waveform import c_ext_L2 ImportError: dlopen(/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so, 0x0002): tried: '/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so' (malformed load command #13 of 15 at 0x1059b45c8 with mh=0x1059b4000, size (0x0) too small), '/System/Volumes/Preboot/Cryptexes/OS/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so' (no such file), '/Users/felix/Documents/software/mtuq_felix/mtuq/mtuq/misfit/waveform/c_ext_L2.cpython-312-darwin.so' (malformed load command #13 of 15 at 0x1059b45c8 with mh=0x1059b4000, size (0x0) too small)

SeismoFelix avatar Apr 05 '25 17:04 SeismoFelix

Hi Felix, Recently, the installation procedure was changed with the addition of "c-compiler" from conda itself. This seemed to help some users, but it could breaking things in your case? You might try experimenting with different compiler settings via the CC environment variable? https://mtuqorg.github.io/mtuq/install/issues.html#cython-compilation-settings

rmodrak avatar Apr 05 '25 18:04 rmodrak

Thanks Ryan,

Could you please elaborate in how the installation prrocedure changed (where was added the c-compiler). I would like (if possible) just to modify the installation file for myself and avoid the conflicts that the new installlation approach is causing to me. I tried to check the modified files in the last commit but I did not see any change related with the installation.

SeismoFelix avatar Apr 05 '25 18:04 SeismoFelix

Sure, here is the change

https://github.com/mtuqorg/mtuq/blob/70f0d55d402e244bdc415fd08ea8409430679414/env_default.yaml#L21

I think it was from about two weeks ago or so.

(I may be away from the computer for a while starting soon)

rmodrak avatar Apr 05 '25 19:04 rmodrak

Thanks Ryan,

I appreciate you dedicate part of your Saturday to this. I simply used an old mtuq/env_default.yaml version where -c-compiler is not included. But did not work:

channels:

  • conda-forge
  • defaults

dependencies:

  • conda-build
  • python=3
  • numpy
  • scipy
  • obspy
  • pandas
  • xarray
  • netCDF4
  • h5py
  • mpi4py
  • retry
  • flake8
  • nose
  • pytest
  • pip
  • pip:
    • git+https://github.com/rmodrak/instaseis.git
    • tables
    • seisgen
    • seishmc
    • -e .

But I am not surprised. I need to become familiar in how to test different compilers settings. I do not know how to this right know, but I will check.

I am also surprised that I did sync my forked version with the master a week ago, before sumitting the pull request, and I could install my mtuq forkerd version for testing the modifications. I could install without any issue. If the installation procedure changed a month ago, then this modification should have affected the installation of my forked version a week ago. But this did not happen.

Thanks,

Felix

SeismoFelix avatar Apr 05 '25 19:04 SeismoFelix

I deleted the environment and the package, then I cloned MTUQ and repeat from scratch the installation process. Before running conda env update --file env.yaml I replaced the env_default.yaml by this old one:

channels:

  • conda-forge
  • defaults

dependencies:

  • conda-build
  • python=3
  • numpy
  • scipy
  • obspy
  • pandas
  • xarray
  • netCDF4
  • h5py
  • mpi4py
  • retry
  • flake8
  • nose
  • pytest
  • pip
  • pip:
    • git+https://github.com/rmodrak/instaseis.git
    • tables
    • seisgen
    • seishmc
    • -e .

And the installation worked. So, in the meantime I will stick with this old file while I figure out what compiler settings use for making the installation worhk with the new env_default.yaml version.

Thanks!

SeismoFelix avatar Apr 05 '25 20:04 SeismoFelix

Hi Felix, Glad to hear the good progress. I will try looking at this from my end as well

rmodrak avatar Apr 06 '25 05:04 rmodrak

Closing now that C extensions have been deprecated in favor of numba (#327)

rmodrak avatar Oct 16 '25 02:10 rmodrak