bioformats icon indicating copy to clipboard operation
bioformats copied to clipboard

OME-TIFF: faster initialization when there are lots of planes

Open melissalinkert opened this issue 3 years ago • 2 comments

Backported from a private PR. Opening to see if repo tests fail, will exclude tomorrow if they do.

This is meant to speed up setId time for cases such as the output of bfconvert "test&sizeZ=4000.fake" test.ome.tiff.

Unit tests fail without 2ebbc58, which I think actually exposes an issue with bfconvert. bfconvert "test&sizeC=2&series=3.fake" -tilex 256 -tiley 256 -channel 1 test-split_%x_%y_%m.ome.tiff results in in this OME-XML:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Warning: this comment is an OME-XML metadata block, which contains crucial dimensional parameters and other important metadata. Please edit cautiously (if at all), and back up the original data before doing so. For more information, see the OME-TIFF web site: https://docs.openmicroscopy.org/latest/ome-model/ome-tiff/. -->
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2016-06" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Creator="OME Bio-Formats 6.10.0-SNAPSHOT" UUID="urn:uuid:aa1ec1c8-4f03-4f22-9634-345a61d7275a" xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2016-06 http://www.openmicroscopy.org/Schemas/OME/2016-06/ome.xsd">
  <Image ID="Image:0" Name="test">
    <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:0" Interleaved="false" SignificantBits="8" SizeC="1" SizeT="1" SizeX="256" SizeY="256" SizeZ="1" Type="uint8">
      <Channel ID="Channel:0:0" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <Channel ID="Channel:0:1" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1">
        <UUID FileName="test-split_0_0_0.ome.tiff">urn:uuid:aa1ec1c8-4f03-4f22-9634-345a61d7275a</UUID>
      </TiffData>
    </Pixels>
  </Image>
  <Image ID="Image:1" Name="test 2">
    <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:1" Interleaved="false" SignificantBits="8" SizeC="1" SizeT="1" SizeX="256" SizeY="256" SizeZ="1" Type="uint8">
      <Channel ID="Channel:1:0" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <Channel ID="Channel:1:1" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
    </Pixels>
  </Image>
  <Image ID="Image:2" Name="test 3">
    <Pixels BigEndian="false" DimensionOrder="XYZCT" ID="Pixels:2" Interleaved="false" SignificantBits="8" SizeC="1" SizeT="1" SizeX="256" SizeY="256" SizeZ="1" Type="uint8">
      <Channel ID="Channel:2:0" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <Channel ID="Channel:2:1" SamplesPerPixel="1">
        <LightPath/>
      </Channel>
      <TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1">
        <UUID FileName="test-split_1_0_1.ome.tiff">urn:uuid:25c6d8cb-ed9d-4f80-99cf-cb4d0e343882</UUID>
      </TiffData>
    </Pixels>
  </Image>
</OME>

That's not introduced by the PR, and fixing is outside the scope, but noting here for discussion.

melissalinkert avatar May 27 '22 16:05 melissalinkert

Excluding, as this will require further work. The changes here are known to cause problems with https://downloads.openmicroscopy.org/images/OME-TIFF/2016-06/BBBC/, so please do not merge.

melissalinkert avatar Sep 02 '22 15:09 melissalinkert

The last two commits here are expected to address known performance problems. Removing exclude label so this can be included in tonight's build.

melissalinkert avatar Sep 07 '22 13:09 melissalinkert

Some additional sanity tests on top of Sebs testing, this time taking a random selection of the remaining public OME-TIFF datasets to ensure no regressions. In all cases the results produced times that were either better or in line with the originals. Overall the results of the PR look to be positive with no notable regressions in any existing datasets.

tubhiswt-4D Leica-2.ome.tiff retina_large.ome.tiff hcs.companion.ome MitoCheck
6.10.1 0.326 0.136 0.683 0.223 0.75
6.10.1 0.311 0.121 0.863 0.22 0.805
6.10.1 0.315 0.132 0.791 0.224 1.195
6.10.1 0.307 0.118 0.666 0.23 0.993
           
bioformats#3831 0.217 0.14 0.625 0.19 0.616
bioformats#3831 0.249 0.146 0.803 0.211 0.808
bioformats#3831 0.262 0.131 0.754 0.216 0.684
bioformats#3831 0.266 0.143 0.733 0.226 0.564

dgault avatar Oct 12 '22 14:10 dgault