scikit-image
Document minimum tifffile requirements
Description:
I'm looking to update the python-scikit-image package in Fedora and getting the following test failures:
_________________________________ test_shapes __________________________________
imgs = <skimage.io.collection.MultiImage object at 0x7fb1f0435790>
def test_shapes(imgs):
imgs = imgs[-1]
assert imgs[0][0].shape == imgs[0][1].shape
> assert imgs[0][0].shape == (10, 10, 3)
E assert (25, 14, 3) == (10, 10, 3)
E At index 0 diff: 25 != 10
E Full diff:
E - (10, 10, 3)
E + (25, 14, 3)
skimage/io/tests/test_multi_image.py:31: AssertionError
---------------------------- Captured stderr setup -----------------------------
Downloading file 'data/multipage_rgb.tif' from 'https://github.com/scikit-image/scikit-image/raw/v0.21.0/skimage/data/multipage_rgb.tif' to '/home/orion/fedora/python-scikit-image/scikit-image-0.21.0/scikit-image/0.21.0'.
_________________________________ test_slicing _________________________________
imgs = [<skimage.io.collection.MultiImage object at 0x7fb20002e190>, <skimage.io.collection.MultiImage object at 0x7fb1f045b8...kimage.io.collection.MultiImage object at 0x7fb1f034ba10>, <skimage.io.collection.MultiImage object at 0x7fb1f03f1c10>]
def test_slicing(imgs):
img = imgs[-1]
assert type(img[:]) is MultiImage
assert len(img[0][:]) + len(img[1][:]) == 26, len(img[:])
assert len(img[0][:1]) == 1
> assert len(img[1][1:]) == 23
E assert 1 == 23
E + where 1 = len(array([[[[6.11859237e-01, 6.13683830e-01, 2.61764427e-01],\n [8.64150092e-01, 1.86032699e-01, 7.67905107e-01],\n [4.47323589e-01, 8.81760559e-01, 7.71544235e-01],\n [3.01772365e-01, 2.93226077e-01, 1.62019224e-01],\n [6.60980691e-01, 9.67724343e-01, 3.53490597e-01],\n [4.16066406e-01, 3.12212669e-01, 5.95506126e-01],\n [1.78595715e-01, 1.74973403e-01, 8.04427253e-01],\n [4.51611405e-01, 3.13848890e-01, 5.25524362e-01],\n [3.05492419e-01, 1.11827285e-02, 9.39158752e-01],\n [2.02804065e-01, 6.23108406e-01, 9.59814407e-01]],\n\n [[4.54140258e-01, 5.17023922e-02, 1.72994131e-01],\n [2.45123764e-01, 5.86041911e-01, 1.44030170e-01],\n [9.49853360e-01, 1.65398332e-01, 9.20412825e-01],\n [9.71664246e-01, 9.13602646e-01, 4.60977608e-01],\n [6.75569279e-01, 7.94274742e-01, 1.72290952e-01],\n [9.94832905e-01, 4.38295464e-01, 6.12733696e-01],\n [1.29133003e-01, 1.69541113e-01, 1.40536150e-02],\n [8.20638267e-01, 4.79702746e-01, 8.87252462e-01],\n [9.30465060e-01, 9.43440274e-02, 1.45653304e-01],\n [4.00729428e-01, 7.57031255e-01, 9.87977575e-01]],\n\n [[4.88...304635e-01]],\n\n [[8.59327355e-01, 4.02425053e-01, 4.76087125e-01],\n [7.08815254e-01, 1.59897390e-01, 6.23051449e-01],\n [9.12360216e-02, 5.42339910e-01, 1.47251478e-01],\n [4.06343227e-01, 1.79473267e-01, 3.69216690e-02],\n [8.84683616e-01, 5.21541897e-01, 9.17095911e-01],\n [1.12484019e-01, 4.47171746e-01, 3.13240591e-01],\n [4.48454609e-01, 4.37365110e-01, 6.00244286e-01],\n [1.49488731e-01, 7.11540362e-01, 8.98939775e-01],\n [6.06455452e-01, 4.72772906e-01, 3.53782699e-01],\n [1.01016308e-01, 6.47838405e-01, 5.74062674e-01]],\n\n [[6.02083416e-01, 5.58439915e-01, 6.48526201e-01],\n [7.07022009e-01, 2.35024425e-01, 1.85182152e-02],\n [6.20179125e-01, 4.63161602e-01, 2.40833443e-01],\n [8.04487714e-03, 4.58802330e-01, 5.27037573e-02],\n [1.33684803e-01, 3.13904377e-01, 5.28891432e-02],\n [4.61955771e-01, 2.37523200e-01, 1.36905619e-01],\n [3.20624017e-02, 2.73770016e-01, 2.19316844e-01],\n [9.68227208e-01, 1.00798813e-01, 8.86375033e-01],\n [5.09561718e-01, 1.48944850e-01, 6.03461718e-01],\n [8.85811169e-02, 6.29287096e-01, 3.73361435e-01]]]]))
skimage/io/tests/test_multi_image.py:43: AssertionError
Way to reproduce:
+ xvfb-run pytest -v --deselect=skimage/data/tests/test_data.py::test_download_all_with_pooch --deselect=skimage/data/tests/test_data.py::test_eagle --deselect=skimage/data/tests/test_data.py::test_brain_3d --deselect=skimage/data/tests/test_data.py::test_cells_3d --deselect=skimage/data/tests/test_data.py::test_kidney_3d_multichannel --deselect=skimage/data/tests/test_data.py::test_lily_multichannel --deselect=skimage/data/tests/test_data.py::test_skin --deselect=skimage/data/tests/test_data.py::test_vortex --deselect=skimage/measure/tests/test_blur_effect.py::test_blur_effect_3d --deselect=skimage/registration/tests/test_masked_phase_cross_correlation.py::test_masked_registration_3d_contiguous_mask skimage
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 -- /usr/bin/python3
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/orion/BUILDROOT/python-scikit-image-0.21.0-1.fc39.x86_64/usr/lib64/python3.11/site-packages/.hypothesis/examples')
rootdir: /home/orion/BUILDROOT/python-scikit-image-0.21.0-1.fc39.x86_64/usr/lib64/python3.11/site-packages
plugins: xprocess-0.22.2, remotedata-0.3.3, asyncio-0.21.0, libtmux-0.21.0, forked-1.6.0, mock-3.10.0, cov-4.0.0, flake8-1.1.1, timeout-2.1.0, rerunfailures-11.0, pyfakefs-5.2.2, xdist-3.3.1, doctestplus-0.12.0, openfiles-0.5.0, anyio-3.5.0, mpi-0.6, hypothesis-6.62.1, localserver-0.7.0, arraydiff-0.5.0
asyncio: mode=Mode.STRICT
collecting ... collected 8215 items / 9 deselected / 2 skipped / 8206 selected
Version information:
3.11.3 (main, May 24 2023, 00:00:00) [GCC 13.1.1 20230511 (Red Hat 13.1.1-2)]
Linux-6.4.0-0.rc2.23.fc39.x86_64-x86_64-with-glibc2.37.9000
numpy version: 1.24.3
are you in a position to list the other python packages installed in your builds? I feel like there have been some changes to imageio that may affect the behavior here.
This should be pretty complete:
python3-3.11.3-2.fc39.x86_64
python3-appdirs-1.4.4-8.fc38.noarch
python3-beniget-0.4.1-6.fc39.noarch
python3-brotli-1.0.9-11.fc38.x86_64
python3-cairo-1.23.0-2.fc38.x86_64
python3-charset-normalizer-3.1.0-1.fc39.noarch
python3-contourpy-1.0.7-2.fc38.x86_64
python3-cycler-0.11.0-7.fc38.noarch
python3-Cython-0.29.34-2.fc39.x86_64
python3-dateutil-1:2.8.2-6.fc39.noarch
python3-devel-3.11.3-2.fc39.x86_64
python3-fonttools-4.39.4-1.fc39.x86_64
python3-fs-2.4.16-2.fc39.noarch
python3-gast-0.5.3-5.fc38.noarch
python3-idna-3.4-2.fc38.noarch
python3-imageio-2.28.1-1.fc39.noarch
python3-iniconfig-1.1.1-11.fc38.noarch
python3-kiwisolver-1.4.4-7.fc38.x86_64
python3-lazy-loader-0.2-2.fc39.noarch
python3-libs-3.11.3-2.fc39.x86_64
python3-lxml-4.9.2-2.fc38.x86_64
python3-markupsafe-2.1.2-1.fc38.x86_64
python3-matplotlib-3.7.1-1.fc39.x86_64
python3-matplotlib-data-3.7.1-1.fc39.noarch
python3-matplotlib-data-fonts-3.7.1-1.fc39.noarch
python3-matplotlib-tk-3.7.1-1.fc39.x86_64
python3-meson-python-0.13.1-3.fc39.noarch
python3-munkres-1.1.2-14.fc38.noarch
python3-networkx-3.1-1.fc39.noarch
python3-numpy-1:1.24.3-1.fc39.x86_64
python3-numpy-f2py-1:1.24.3-1.fc39.x86_64
python3-olefile-0.46-19.fc38.noarch
python3-packaging-23.1-1.fc39.noarch
python3-pillow-9.5.0-1.fc39.x86_64
python3-pillow-tk-9.5.0-1.fc39.x86_64
python3-pip-23.1.2-1.fc39.noarch
python3-pluggy-1.0.0-6.fc38.noarch
python3-ply-3.11-18.fc38.noarch
python3-pooch-1.5.2-10.fc38.noarch
python3-pyparsing-3.0.9-7.fc39.noarch
python3-pyproject-metadata-0.7.1-1.fc38.noarch
python3-pysocks-1.7.1-17.fc39.noarch
python3-pytest-7.3.1-2.fc39.noarch
python3-pytest-localserver-0.7.0-2.fc38.noarch
python3-pywt-1.3.0-3.fc38.x86_64
python3-requests-2.28.2-2.fc39.noarch
python3-rpm-generators-14-6.fc39.noarch
python3-rpm-macros-3.11-10.fc39.noarch
python3-scipy-1.10.1-2.fc39.x86_64
python3-setuptools-67.7.2-2.fc39.noarch
python3-six-1.16.0-9.fc38.noarch
python3-tifffile-2020.7.4-10.fc38.noarch
python3-tkinter-3.11.3-2.fc39.x86_64
python3-urllib3-1.26.12-4.fc39.noarch
python3-werkzeug-2.2.3-1.fc39.noarch
Have you run skimage on this system before? Can you try cleaning out the skimage directory in ~/.cache?
So, I'm doing some slightly odd things to support the RPM build in the Fedora builders which do not have network access. First I'm running in a freshly unpacked directory with XDG_CACHE_DIR=$PWD - so it does do a fresh download of all of the images. Then I tar up the scikit-image/0.21.0 directory for use on the builders. But the initial build is done with an empty cache. And you can see the Downloading output in the test stderr output.
Hmm, but interestingly enough now I'm not seeing the test failures in the actual package build in the mock chroot. So something extra in my dev environment must be triggering the issue.
Just as a note, we know that there have been some changes in how multi image files expose the images.
at scikit-image, we dispatch image loading to many other libraries:
- tifffile
- imageio
- which in turn may dispatch to pillow.
Those would be the places I would first look for additional differences.
Looks like tifffile is quite out of date in Fedora - working on updating that now too. Thanks.
hint: tifffile changed alot in the last two years moving much of the code to a package called imagecodecs
I help packge tifffile for conda-forge so feel free to ask questions: https://github.com/conda-forge/tifffile-feedstock/blob/main/recipe/meta.yaml
if you hit any trouble.
Thanks again - looks like I am going to need to package up imagecodecs for Fedora.
eek, yeah that was a tough transition for conda-forge as well.
ALOT of the features of imagecodecs are optional.
We have worked with cgohlke to help us understand the different options.
He eventually helped us upstream our customizations to not drag around patches: https://github.com/cgohlke/imagecodecs/blob/master/setup.py#L458
Goodluck!
I'm reopening this issue as a documentation issue. Hopefully we can help communicate these issues to our end users better.
I expect you won't be the last one to hit this.
Hello scikit-image core devs! There hasn't been any activity on this issue for more than 180 days. I have marked it as "dormant" to make it easy to find. To our contributors, thank you for your contribution and apologies if this issue fell through the cracks! Hopefully this ping will help bring some fresh attention to the issue. If you need help, you can always reach out on our forum If you think that this issue is no longer relevant, you may close it, or we may do it at some point (either way, it will be done manually).
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I'm working on deprecating our plugin infrastructure. Having removed that, the only other occurrence where import tifffile directly is currently skimage.io.ImageCollection. So long-term we might be able to address this by no longer depending on tifffile at all. Also, follow this forum discussion for updates on that.
Hum, so I'm another Fedora packager working on the same package, and this seems to be back again:
=================================== FAILURES ===================================
_________________________________ test_shapes __________________________________
imgs = <skimage.io.collection.MultiImage object at 0xffff6ca0ef70>
def test_shapes(imgs):
imgs = imgs[-1]
assert imgs[0][0].shape == imgs[0][1].shape
> assert imgs[0][0].shape == (10, 10, 3)
E assert (25, 14, 3) == (10, 10, 3)
E At index 0 diff: 25 != 10
E Full diff:
E - (10, 10, 3)
E + (25, 14, 3)
skimage/io/tests/test_multi_image.py:36: AssertionError
_________________________________ test_slicing _________________________________
imgs = [<skimage.io.collection.MultiImage object at 0xffff6c48afd0>, <skimage.io.collection.MultiImage object at 0xffff6c48b1...kimage.io.collection.MultiImage object at 0xffff6c48ba70>, <skimage.io.collection.MultiImage object at 0xffff6c48b4d0>]
def test_slicing(imgs):
img = imgs[-1]
assert type(img[:]) is MultiImage
assert len(img[0][:]) + len(img[1][:]) == 26, len(img[:])
assert len(img[0][:1]) == 1
> assert len(img[1][1:]) == 23
E assert 1 == 23
E + where 1 = len(array([[[[6.11859237e-01, 6.13683830e-01, 2.61764427e-01],\n [8.64150092e-01, 1.86032699e-01, 7.67905107e-01],\n [4.47323589e-01, 8.81760559e-01, 7.71544235e-01],\n [3.01772365e-01, 2.93226077e-01, 1.62019224e-01],\n [6.60980691e-01, 9.67724343e-01, 3.53490597e-01],\n [4.16066406e-01, 3.12212669e-01, 5.95506126e-01],\n [1.78595715e-01, 1.74973403e-01, 8.04427253e-01],\n [4.51611405e-01, 3.13848890e-01, 5.25524362e-01],\n [3.05492419e-01, 1.11827285e-02, 9.39158752e-01],\n [2.02804065e-01, 6.23108406e-01, 9.59814407e-01]],\n\n [[4.54140258e-01, 5.17023922e-02, 1.72994131e-01],\n [2.45123764e-01, 5.86041911e-01, 1.44030170e-01],\n [9.49853360e-01, 1.65398332e-01, 9.20412825e-01],\n [9.71664246e-01, 9.13602646e-01, 4.60977608e-01],\n [6.75569279e-01, 7.94274742e-01, 1.72290952e-01],\n [9.94832905e-01, 4.38295464e-01, 6.12733696e-01],\n [1.29133003e-01, 1.69541113e-01, 1.40536150e-02],\n [8.20638267e-01, 4.79702746e-01, 8.87252462e-01],\n [9.30465060e-01, 9.43440274e-02, 1.45653304e-01],\n [4.00729428e-01, 7.57031255e-01, 9.87977575e-01]],\n\n [[4.88...304635e-01]],\n\n [[8.59327355e-01, 4.02425053e-01, 4.76087125e-01],\n [7.08815254e-01, 1.59897390e-01, 6.23051449e-01],\n [9.12360216e-02, 5.42339910e-01, 1.47251478e-01],\n [4.06343227e-01, 1.79473267e-01, 3.69216690e-02],\n [8.84683616e-01, 5.21541897e-01, 9.17095911e-01],\n [1.12484019e-01, 4.47171746e-01, 3.13240591e-01],\n [4.48454609e-01, 4.37365110e-01, 6.00244286e-01],\n [1.49488731e-01, 7.11540362e-01, 8.98939775e-01],\n [6.06455452e-01, 4.72772906e-01, 3.53782699e-01],\n [1.01016308e-01, 6.47838405e-01, 5.74062674e-01]],\n\n [[6.02083416e-01, 5.58439915e-01, 6.48526201e-01],\n [7.07022009e-01, 2.35024425e-01, 1.85182152e-02],\n [6.20179125e-01, 4.63161602e-01, 2.40833443e-01],\n [8.04487714e-03, 4.58802330e-01, 5.27037573e-02],\n [1.33684803e-01, 3.13904377e-01, 5.28891432e-02],\n [4.61955771e-01, 2.37523200e-01, 1.36905619e-01],\n [3.20624017e-02, 2.73770016e-01, 2.19316844e-01],\n [9.68227208e-01, 1.00798813e-01, 8.86375033e-01],\n [5.09561718e-01, 1.48944850e-01, 6.03461718e-01],\n [8.85811169e-02, 6.29287096e-01, 3.73361435e-01]]]]))
skimage/io/tests/test_multi_image.py:50: AssertionError
This is building with tifffile 2024.5.3-2.fc41 , without imagecodecs installed at all. If I add imagecodecs to the build root, I get more failures, all caused by a ValueError: <COMPRESSION.ADOBE_DEFLATE: 8> requires the 'imagecodecs' package (I guess our imagecodecs is missing whatever that wants?), and those two failures don't go away.
oh, as for the other package versions, they are:
- python3-pillow 10.3.0-2.fc41
- python3-imageio 2.34.1-2.fc41
those both seem to be the latest.
ahhhh, I figured this out: it's entirely a downstream issue. no_time_for_that_tiny.gif - which these tests use - is not in the cached data tarball we use to avoid a network lookup (see https://github.com/scikit-image/scikit-image/issues/6993#issuecomment-1575357931 ). it just needs adding to that tarball, then the tests pass.
Hmm, it does seem like no_time_for_that_tiny.gif does not end up in the cache directory when running the package build locally. Why would that be?
Looks like that gif is missing from the legacy dataset, even though it does exist in the github repo: https://github.com/scikit-image/scikit-image/blob/main/skimage/data/_registry.py#L28
How do you generate your offline data-bundle? Presumably, you zip up the files in the data dir? In that case, I think simply adding the gif to the legacy list should fix things.