OMERO export fails to create OME-TIFF as BigTIFF

[aherbert]

Export of the following size image fails:

X=1080
Y=1080
Z=6
C=2
T=144
pixel type = unsigned 16-bit

Size = 1080x1080x6x2x144 x2 = 4,031,078,400 bytes

When exporting through OMERO Insight or using the omero-py CLI there is an error as the combined pixel size and the metadata is above the limit for a TIFF (2^32 = 4,294,967,296):

Relevant stack trace from Insight:

serverStackTrace = "loci.formats.FormatException: File is too large; call setBigTiff(true)
                         at loci.formats.out.TiffWriter.prepareToWriteImage(TiffWriter.java:385)
                         at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:272)
                         at loci.formats.out.OMETiffWriter.saveBytes(OMETiffWriter.java:219)
                         at loci.formats.out.TiffWriter.saveBytes(TiffWriter.java:212)
                         at ome.services.blitz.impl.ExporterI$2.doWork(ExporterI.java:414)

From omero-py:

omero export --file tmp.ome.tif Image:1430702
...
InternalException: File is too large; call setBigTiff(true)

The offending code that raises the exception appears to be here:

    // loci.formats.out.TiffWriter.prepareToWriteImage(...)
    if (!isBigTiff) {
      isBigTiff = (out.length() + 2
          * (width * height * c * bytesPerPixel)) >= 4294967296L;
      if (isBigTiff) {
        throw new FormatException("File is too large; call setBigTiff(true)");
      }
    }

However it is the code that calls this writer than needs to set the BigTIFF option to enabled as auto detection is not working given the extra metadata size.

The image is one of a plate obtained from a Operetta plate reader. I did not check the BioFormats code to determine whether the export is including the XML for the entire plate in the OME-TIFF. There is enough metadata to exceed the limit for a TIFF.

[sbesson]

Thanks @aherbert for the report.

This is another occurence of the writing bug described in https://github.com/ome/bioformats/issues/4113. Shortly the criteria for setting the BigTIFF in the ImageWriter initialization is inconsistent with the internal checks in the TiffWriter.prepareToWriteImage.

In the case of the OMERO exporter, the relevant logic when initializing the reader is in https://github.com/ome/omero-blitz/blob/b56a31dff4a7470a1b1947e6e9cd9019642480ee/src/main/java/ome/services/blitz/impl/ExporterI.java#L384-L388. This suggests in your case, the total size did not exceed the 4GiB limit for setting the BigTIFF file after adding the metadata size.

When using the Bio-Formats command-line tools, a workaround is to force the BigTIFF flag by using an extension like ome.btf or ome.tf8. This will not work with the exporter as the filename is internally set in https://github.com/ome/omero-blitz/blob/b56a31dff4a7470a1b1947e6e9cd9019642480ee/src/main/java/ome/services/blitz/impl/ExporterI.java#L370-L371 and uses the ome.tiff extension.

The resolution of this export issue will require a Bio-Formats fix. A workaround might be to either download the original data or access it directly and use converter tools like bfconvert or bioformats2raw + raw2ometiff directly.

[aherbert]

Our current workaround is to export the image pixels using the Python API to a numpy array and then save it using the tifffile library.

We will wait patiently for direct support from OMERO export.