Img Writer: Writing Tiffs is _very_ slow

[dietzc]

Writing a tiff with e.g. 1024,1024 and Z=51, Channel = 3 in this dimension order is super slow. One reason might be, that it seems that nearly every pixel write action is logged somewhere. @gab1one please reproduce and double check!

[gab1one]

Yes, the tiff writers (tiff and ome.tiff) are about 6.000% slower than eq. formats like ics. Even with small images. One example:

• SCIFIO can write the same image (128,128, C=1 Z=1, T=50, ) 137.165 ± 2.115 times a second as ICS file but only about 2.093 ± 0.063 a second as (ome.)tiff file.
  take a look at the benchmark results

I removed logging from the TIFF format to check your hypothesis but it did not produce any meaningful performance improvement (up to 0.4 more operations more per second!). without logging

Larger files take much longer to write but show the same effect.

[gab1one]

see https://github.com/knime-ip/knip/commit/46b6799b83512270b9752ec555cef5cbdf95528b for the first big improvement. Writing the 160mb file sequentially now takes about 10 seconds instead of the 44 it took before

[dietzc]

great work @gab1one. Reading & Writing Tiffs (and also other formats) is very often used by the users and therefore performance is crucial.

[gab1one]

Further improvements depend on changes in SCIFIO, I will talk with Mark about that.

[imagejan]

Any news on this? I'm still suffering severe performance issues with the Image Writer.

Here's an example benchmark workflow: Image Writer Performance.zip

I get the following writing times for the different file formats:

Node 2    tif       254156 

Node 3    ome.tif   191286 

Node 4    ics       549 

[dietzc]

I think this is still a SCIFIO Problem. At the moment I can only recommend to use ics in favour of tiff. @ctrueden what do you think?

[ctrueden]

I am aware of a SCIFIO issue for improving the performance of TIFF reading: scifio/scifio#7.

But is there an issue for improving the behavior of TIFF writing? I couldn't find it.

Has anyone profiled why TIFF writing is slow? I can't imagine it would be that hard to fix. #famouslastwords

[dietzc]

@gab1one can you reproduce the problem and open an issue in scifio?

[imagejan]

The slowness issue is reproducible in ImageJ, the relevant SCIFIO issue is https://github.com/scifio/scifio/issues/310.

[dietzc]

see also: https://tech.knime.org/forum/knime-image-processing/image-writer-cannot-write-to-network-drives#comment-48710

[tibuch]

Any news regarding tif writing speed? I am currently using the image writer to move my data to the python-nodes and it takes a very long time.

[ctrueden]

@tibuch Sorry for the long delay in reply. Unfortunately, no one is working on TIFF performance. I am very sorry this has sat for so long, but the fact is that we still do not have a dedicated SCIFIO developer who has time to prioritize this issue... yet.