Enable testing on a big-endian image

[joshmoore]

trafficstars

Uses qemu-user-static to build on a big-endian system in order to test #844

TODO:

• [ ] Add unit tests and/or doctests in docstrings
• [ ] Add docstrings and API docs for any new/modified user-facing classes and functions
• [ ] New/modified features documented in docs/tutorial.rst
• [ ] Changes documented in docs/release.rst
• [ ] GitHub Actions have all passed
• [ ] Test coverage is 100% (Codecov passes)

[joshmoore]

Build showing various errors (though not the specific one from #844):

FAILED zarr/tests/test_core.py::TestArrayWithN5FSStore::test_nchunks_initialized
FAILED zarr/tests/test_core.py::TestArrayWithCustomMapping::test_nbytes_stored
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_append_2d
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_append_2d_axis
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_1d
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_1d_fill_value
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_1d_selections
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_2d
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_2d_edge_case
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_2d_partial
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_order
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_islice
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_iter - ...
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_non_cont
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_np_ufuncs
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_read_from_all_blocks
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_read_nitems_less_than_blocksize_from_multiple_chunks
FAILED zarr/tests/test_core.py::TestArrayWithFSStorePartialRead::test_resize_2d

Several are of the form:

>       b = f.read(length)
E       ValueError: read length must be non-negative or -1

/usr/lib/python3/dist-packages/fsspec/utils.py:261: ValueError

Trying to upgrade fsspec first.

[joshmoore]

Updated the branch today. Another phenotype in case it gives anyone any clues:

=================================== FAILURES ===================================
________________ TestArrayWithCustomMapping.test_nbytes_stored _________________

self = <zarr.tests.test_core.TestArrayWithCustomMapping testMethod=test_nbytes_stored>

    def test_nbytes_stored(self):
        z = self.create_array(shape=1000, chunks=100)
        assert 245 == z.nbytes_stored
        z[:] = 42
>       assert 515 == z.nbytes_stored
E       AssertionError: assert 515 == 495
E         +515
E         -495

zarr/tests/test_core.py:2402: AssertionError

[avalentino]

Are there news on the issue with test_nbytes_stored? We also see it on Debian: https://ci.debian.net/data/autopkgtest/testing/s390x/r/resampy/26735584/log.gz

[joshmoore]

Hey @avalentino. No, no new news since I assumed I was doing something wrong with this PR. Very much thanks for the heads up that this is in the wild!

[joshmoore]

Re-enabled GHA after https://github.com/zarr-developers/community/issues/44 but now need to fix:

E: The repository 'http://ports.ubuntu.com/ubuntu-ports hirsute-security Release' does not have a Release file.

https://fridge.ubuntu.com/2022/01/21/ubuntu-21-04-hirsute-hippo-end-of-life-reached-on-january-20-2022/

Moving to focal (20.04LTS).

[jakirkham]

Think it should be possible to use the setup-miniconda workflow used in other jobs for aarch64 & ppc64le. This might save the pains around figuring out a whole new workflow for these architectures.

[joshmoore]

@jakirkham: that would work inside of the docker?

[jakirkham]

Raised issue ( https://github.com/conda-incubator/setup-miniconda/issues/247 ) on aarch64 & ppc64le, but missed we are using s390x, which is neither of those 🤦‍♂️

conda-forge does similar things to what we are doing here for some feedstocks. There does appear to be a Miniconda installer for s390x.

In terms of Numcodecs, we could either compile it from source here or enable s390x wheel building on the Numcodecs repo, which we could then install. Had disabled the latter previously as it was taking too much time on CI, but we could revisit (given there is a need).

[avalentino]

FYI: after the update to zarr v2.17 we started observing the following additional failures on s390x:

179s FAILED tests/test_core.py::TestArrayWithN5Store::test_hexdigest - AssertionEr...
179s FAILED tests/test_core.py::TestArrayWithN5FSStore::test_hexdigest - Assertion...
179s FAILED tests/test_core.py::TestArrayWithCustomMapping::test_nbytes_stored - a...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_1d - V...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_1d_fill_value
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_1d_selections
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_2d - V...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_2d_edge_case
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_2d_partial
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_array_order
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_resize_2d - ...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_append_2d - ...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_append_2d_axis
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_np_ufuncs - ...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_iter - OSErr...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_islice - OSE...
179s FAILED tests/test_core.py::TestArrayWithFSStorePartialRead::test_non_cont - O...

The full log is at https://ci.debian.net/packages/z/zarr/testing/s390x/43459778/

[joshmoore]

Thanks for the heads up, @avalentino. I don't have any new or immediate insights, but perhaps we can make a wider call for help, @MSanKeys963.