No binaries for pip install on aarch64

[sgofferj]

What happened?

I'm building a docker container for aarch64. In the build I call pip install eccodes cfgrib. The python packages are installed but python -m cfgrib selfcheck fail with RuntimeError: Cannot find the ecCodes library. On detailed investigation I cannot find any libeccodes binaries anywhere.

What are the steps to reproduce the bug?

Try to pip install eccodes on an aarch64 system

Version

2.38

Platform (OS and architecture)

Debian Bookworm (Linux 5.4.17-2136.324.5.3.el8uek.aarch64 #2 SMP Tue Oct 10 16:49:45 PDT 2023 aarch64 GNU/Linux)

Relevant log output

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/usr/local/lib/python3.12/site-packages/cfgrib/__init__.py", line 20, in <module>
    from .cfmessage import COMPUTED_KEYS
  File "/usr/local/lib/python3.12/site-packages/cfgrib/cfmessage.py", line 29, in <module>
    from . import abc, messages
  File "/usr/local/lib/python3.12/site-packages/cfgrib/messages.py", line 28, in <module>
    import eccodes  # type: ignore
    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/eccodes/__init__.py", line 13, in <module>
    from .eccodes import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/eccodes/eccodes.py", line 12, in <module>
    from gribapi import (
  File "/usr/local/lib/python3.12/site-packages/gribapi/__init__.py", line 13, in <module>
    from .gribapi import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gribapi/gribapi.py", line 34, in <module>
    from gribapi.errors import GribInternalError
  File "/usr/local/lib/python3.12/site-packages/gribapi/errors.py", line 16, in <module>
    from .bindings import ENC, ffi, lib
  File "/usr/local/lib/python3.12/site-packages/gribapi/bindings.py", line 105, in <module>
    raise RuntimeError("Cannot find the ecCodes library")
RuntimeError: Cannot find the ecCodes library

Accompanying data

No response

Organisation

No response

[sgofferj]

So, during the docker build, pip does download a wheel Downloading eccodes-2.38.0-py3-none-any.whl (43 kB)

It builds the wheels for cartopy and findlibs (but not eccodes) and then reports the install finished

Building wheels for collected packages: cartopy, findlibs

[...]

Installing collected packages: pytz, findlibs, appdirs, urllib3, tzdata,
typing-extensions, traitlets, six, pyshp, pyparsing, pycparser,
platformdirs, pillow, packaging, numpy, kiwisolver, idna, fonttools,
cycler, click, charset-normalizer, certifi, attrs, shapely, scipy, requests,
python-dateutil, pyproj, flexparser, flexcache, contourpy, cffi,
pooch, pint, pandas, matplotlib, eccodes, xarray, cfgrib, cartopy, metpy

[ztzthu]

I had the same problem. How did you solve it in the end ? Did you install the library by youself?

[sgofferj]

I had the same problem. How did you solve it in the end ? Did you install the library by youself?

Yeah, I included compiling and installation in my Dockerfile. It's a two-stage build.

Builder:

[...]
RUN apt-get -y update \
    && apt-get -y --no-install-recommends install build-essential gfortran libaec-dev cmake wget python3-pip \
    && rm -rf /root/.cache \
    && rm -rf /var/lib/apt/lists/*

RUN wget https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.38.0-Source.tar.gz
RUN tar xzf eccodes-2.38.0-Source.tar.gz && rm eccodes-2.38.0-Source.tar.gz
RUN mkdir build
WORKDIR /app/build
RUN cmake -DCMAKE_INSTALL_PREFIX=/usr/local ../eccodes-2.38.0-Source
RUN make
RUN ctest
[...]

Image:

[...]
COPY --from=builder /app/build/lib/libeccodes* /usr/local/lib/
COPY --from=builder /app/build/bin/* /usr/local/bin/
COPY --from=builder /app/eccodes-2.38.0-Source/definitions /usr/local/share/eccodes/definitions
[...]

[amarandon]

Which version of Python are you using @sgofferj ?

Here I observed that it works with Python 3.9 and Python 3.12 but not with Python 3.13. It seems that the logic that resolves the binary library doesn't work with the latest version of Python.

EDIT: OK just saw your Python version in the stackstrace, so not working with 3.12 for you either :thinking: EDIT2: interestingly I'm observing the same difference between versions of Python on Windows

[frazane]

I have the same problem for version >=2.39.0. For versions 2.37.0, 2.38.1 and 2.38.3 I have no problems.

[lscheilling]

I too found it's broken on versions >=2.39.0, seems to be picking up the wrong wheel

Broken: eccodes-2.39.1-py3-none-any.whl

Working: eccodes-2.38.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

If you try to install the manylinux wheel manually you will get the error: ERROR: eccodes-2.39.1-cp39-cp39-manylinux_2_28_x86_64.whl is not a supported wheel on this platform.

Edit: Looks like my issue was related to this commit https://github.com/ecmwf/eccodes-python/commit/f19e22e61172e677f49a2f3a9b2a462c18002100, now requires glibc 2.28 or later (was previously glibc 2.17)

[dschneider-wxs]

i was not able to pip install any version >=2.37.0 using python:3.11-slim docker image. I resorted to manually installing the library . Thanks @sgofferj !

[MHBalsmeier]

@shahramn Would it be possible to provide Aarch64 non-MACOSX wheels here? https://pypi.org/project/eccodes/2.41.0/#files

Quite a few people seem to have this problem.