ornladios
Read_attribute broken with numpy > 2.0.0 on redhat and 2.1.1 on ubuntu
I have the following minimal script that writes a numpy array as an attribute in adios2, and in turn read it into the code again:
import adios2.bindings as adios2
from mpi4py import MPI
import numpy as np
engine = "BP5"
filename = "test.bp"
adios = adios2.ADIOS(MPI.COMM_WORLD)
io = adios.DeclareIO("writer")
out_data = np.array([13, 11, 7, 5], dtype=np.int32)
# Write array to file as attribute
io.SetEngine(engine)
adios_file = io.Open(str(filename), adios2.Mode.Write, MPI.COMM_WORLD)
adios_file.BeginStep()
io.DefineAttribute("data", out_data)
adios_file.PerformPuts()
adios_file.EndStep()
adios_file.Close()
adios.RemoveIO("writer")
# Read array from file as attribute
io = adios.DeclareIO("reader")
io.SetEngine(engine)
adios_file = io.Open(str(filename), adios2.Mode.Read, MPI.COMM_WORLD)
adios_file.BeginStep()
attr = io.InquireAttribute("data")
in_data = attr.Data()
adios_file.EndStep()
adios_file.Close()
adios.RemoveIO("reader")
np.testing.assert_array_equal(out_data, in_data)
This works fine with rockylinux 9:
FROM rockylinux/rockylinux:9
ARG ADIOS2_VERSION=v2.10.1
ARG NUMPY_VERSION=1.26.4
ARG MPICH_VERSION=4.2.2
WORKDIR /tmp
RUN dnf -y update && \
dnf install -y dnf-plugins-core && \
dnf config-manager --set-enabled crb && \
dnf install -y epel-release && \
dnf -y install \
cmake \
gcc \
gcc-c++ \
gcc-gfortran \
pugixml-devel \
python3 \
python3-devel \
python3-pip \
spdlog-devel \
# Utility
git \
# Only in crb set
ninja-build \
openblas-devel && \
dnf -y clean all && \
rm -rf /var/cache
# Build MPICH (see https://github.com/pmodels/mpich/issues/5811)
RUN curl -L -O https://www.mpich.org/static/downloads/${MPICH_VERSION}/mpich-${MPICH_VERSION}.tar.gz && \
tar -xf mpich-${MPICH_VERSION}.tar.gz && \
cd mpich-${MPICH_VERSION} && \
FCFLAGS=-fallow-argument-mismatch FFLAGS=-fallow-argument-mismatch ./configure --with-device=ch4:ofi --enable-shared --prefix=/usr/local && \
make -j 4 install && \
rm -rf /tmp/*
# Install numpy and Mpi4py
RUN python3 -m pip install numpy==${NUMPY_VERSION} mpi4py
# BUILD ADIOS2
RUN git clone --branch=${ADIOS2_VERSION} --single-branch https://github.com/ornladios/ADIOS2.git adios2 && \
cmake -G Ninja -DADIOS2_USE_HDF5=off -DADIOS2_USE_Python=on -DADIOS2_USE_Fortran=off -DBUILD_TESTING=off -DADIOS2_BUILD_EXAMPLES=off -DADIOS2_USE_ZeroMQ=off -B build-dir-adios2 -S ./adios2 && \
cmake -G Ninja -B build-dir-adios2 -DCMAKE_BUILD_TYPE="Release" -S ./adios2/ && \
cmake --build build-dir-adios2 --parallel 2 && \
cmake --install build-dir-adios2
WORKDIR /src/test/
COPY mwe.py mwe.py
RUN python3 mwe.py
RUN bpls -a -l test.bp
which you can build with:
docker build -t adios2_image --build-arg="NUMPY_VERSION=1.26.4" . --progress=plain
However, using numpy 2.0.1, one gets
docker build -t adios2_image --build-arg="NUMPY_VERSION=2.0.1" . --progress=plain
with the following traceback
> [ 9/10] RUN python3 mwe.py:
1.777 File "/usr/local/lib64/python3.9/site-packages/numpy/testing/_private/utils.py", line 885, in assert_array_compare
1.777 raise AssertionError(msg)
1.777 AssertionError:
1.777 Arrays are not equal
1.777
1.777 Mismatched elements: 3 / 4 (75%)
1.777 Max absolute difference among violations: 8
1.777 Max relative difference among violations: 0.61538462
1.777 ACTUAL: array([13, 11, 7, 5], dtype=int32)
1.777 DESIRED: array([13, 13, 13, 13], dtype=int32)
------
Dockerfile:55
--------------------
53 | WORKDIR /src/test/
54 | COPY mwe.py mwe.py
55 | >>> RUN python3 mwe.py
56 |
57 | RUN bpls -a -l test.bp
--------------------
ERROR: failed to solve: process "/bin/sh -c python3 mwe.py" did not complete successfully: exit code: 1
This is now present on ubuntu:24.04 with numpy==2.1.1.
Following is a minimal dockerfile (using the same code as above to verify the Python code):
FROM ubuntu:24.04
ARG MPI="openmpi"
ENV OPENBLAS_NUM_THREADS=1 \
OPENBLAS_VERBOSE=0
ENV DEB_PYTHON_INSTALL_LAYOUT=deb_system
ENV DEBIAN_FRONTEND=noninteractive
ARG ADIOS2_VERSION=v2.10.0
WORKDIR /tmp
RUN apt-get -qq update && \
apt-get -yq upgrade && \
apt-get -y install \
lib${MPI}-dev \
python3-dev \
python3-pip \
python3-setuptools \
python3-venv \
git \
cmake \
ninja-build \
build-essential && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
ENV VIRTUAL_ENV=/test-env
ENV PATH=${VIRTUAL_ENV}/bin:$PATH
RUN python3 -m venv ${VIRTUAL_ENV}
# Install Python packages (via pip)
ARG NUMPY_VERSION=2.1.1
RUN python3 -m pip install --no-cache-dir mpi4py numpy==${NUMPY_VERSION}
ENV CMAKE_INSTALL_PYTHONDIR=${VIRTUAL_ENV}/lib/python3.12/site-packages
# BUILD ADIOS2
RUN git clone --branch=${ADIOS2_VERSION} --single-branch https://github.com/ornladios/ADIOS2.git adios2 && \
cmake -G Ninja -DADIOS2_USE_HDF5=off -DADIOS2_USE_Python=on -DADIOS2_USE_Fortran=off -DBUILD_TESTING=off -DADIOS2_BUILD_EXAMPLES=off -DCMAKE_INSTALL_PYTHONDIR=${CMAKE_INSTALL_PYTHONDIR} -DADIOS2_USE_ZeroMQ=off -B build-dir-adios2 -S ./adios2 && \
cmake -G Ninja -B build-dir-adios2 -DCMAKE_BUILD_TYPE="Release" -S ./adios2/ && \
cmake --build build-dir-adios2 --parallel 2 && \
cmake --install build-dir-adios2
ENV LD_LIBRARY_PATH=/usr/local/lib
WORKDIR /src/test
COPY mwe.py mwe.py
RUN python3 mwe.py -v
RUN bpls -a -l test.bp
Note that this has seems to have been fixed on the master branch of ADIOS2, so it would be good with a new release.
Thank you for the report. I can reproduce this on ubuntu 24.04 and adios2 release_210 branch. And as you state, the master branch works fine.
I have the same issue. Would it be possible to create a new release of adios2?
We decided to make a 2.10.2 release with a handful of bug fixes, including this one. Stay tuned.
We decided to make a 2.10.2 release with a handful of bug fixes, including this one. Stay tuned.
Any update on this?
Yes, it was delayed by vacation, but we're hoping to have 2.10.2 out by the end of the week.
