Improve ImageJ's handling of large (>4GB) TIFF files

[ctrueden]

It should be possible to quickly load, browse and save very large TIFF files in a coherent way. There are several pieces to this:

• [ ] Ensure SCIFIO transparently writes to BigTIFF when exporting using File > Export > Image...
• [ ] Make SCIFIO able to import ImageJ1's "raw pseudo TIFF" format
• [ ] Make SCIFIO's "virtual stack" (cell image) performance as good as possible
• [ ] Make SCIFIO recognize signs (e.g., an ImageJ1-style ImageDescription in the first IFD) when a TIFF is written monotonically, so it can be parsed more quickly (i.e., without explicitly reading all the IFDs)
• [ ] Make SCIFIO add such signs (probably some new metadata as a directory entry in the first IFD) when writing TIFFs

Also critical is to improve performance of Bio-Formats setId calls of various high-impact proprietary file formats such as ND2, although that work is outside the scope of this issue.

See also these mailing list threads:

• Problems with Saving Tiff greater than 4 GB & bigTiff exporter
• Processing huge files
• bfconverter nd2 to tif

[tinevez]

What is a telltale?

[ctrueden]

@tinevez I edited the description to use the word "signs" instead of "telltales" and gave a couple of parenthetical examples.

[ehrenfeu]

@tinevez "telltales" are thin threads helping you in trimming your sails: http://www.startedsailing.com/tell-tales.html I'm not sure how this is related in improving the SCIFIO/Bio-Formats performance, but if sailing helps I'd be more than happy to take everyone involved out for a cruise :boat: :anchor: :wink:

[tischi]

Hi All,

Just as an information: I am currently testing different file formats for opening and saving >4GB 5D image files in ImageJ and Imaris and for me ics works very well (can be exported with Bioformats).

https://en.wikipedia.org/wiki/Image_Cytometry_Standard

I am mentioning this here as I feel this is a underrated image type that might deserve more attention.

[ctrueden]

Thanks @tischi, ICS is indeed a nice format in many ways.

[uxhub]

Hi all,

Finally, is there a simple way to save big tiff file (>4gb) that will not produce a message like "Stack is larger than 4GB. Most TIFF readers will only open the first image. Use this information to open as raw: [...]" (file>save as>tiff), or like "loci.formats.FormatException: File is too large; call setBigTiff(true)" by using (plugins > bio-formats > bio-formats exporter) ?

Many thanks

[ctrueden]

@uxhub This issue is about addressing precisely that limitation: there is no simple way currently. When this issue closes, it will be because the situation has improved, and saving large TIFFs "just works" as expected.

[tischi]

@uxhub : why don't you use ICS? I think it is better than .tif in any aspect that I can think of.

[uxhub]

OK thank you for your replies. I will have a look on ICS format, but it doesn't seem to be standard in a lot of API and software. (I'll probably use the raw format...). Could libtiff and bigtiff be used in IJ to save properly large tiff files ? Regards,

[tischi]

to me ICS seems to be the only standard worth talking of at the moment. works with Imaris, ImageJ, Arivis, Matlab, Python, anything that uses BioFormats anyway.

[ctrueden]

The only problem with ICS is that it will lose a lot of metadata compared to OME-TIFF. So we do really need to fix these limitations with respect to 4+GB TIFF & OME-TIFF files.