aicsimageio feature/tiledtiffwriter

OmeTiledTiffWriter

This PR adds adds a writer to wrap the bfio.BioWriter(backend="python"). This writer will only write OME TIFFs with the tile width and tile tags set to 1024 and dimension order XYZCT with Deflate compression. The BioWriter performs pseudo-threaded writing, where compression is performed in threads and written in the main thread as they complete. This leads to significant performance improvements for images larger than 1024x1024, or images of moderate size with many image planes. The potential deficit to this format is that image chunks may be scattered throughout the file, potentially leading to slower reading times on older hard drives. However, early testing shows marginal performance deficits, but thorough performance testing has not been performed.

This writer will create images that are compatible with the OmeTiledTiffReader.

Apr 11 '22 13:04 Nicholas-Schaub

Codecov Report

Patch coverage: 93.02% and project coverage change: +0.29 :tada:

Comparison is base (8b44ff7) 93.71% compared to head (65ff1cf) 94.01%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #396      +/-   ##
==========================================
+ Coverage   93.71%   94.01%   +0.29%     
==========================================
  Files          47       50       +3     
  Lines        3580     4343     +763     
==========================================
+ Hits         3355     4083     +728     
- Misses        225      260      +35

Impacted Files	Coverage Δ
aicsimageio/formats.py	`100.00% <ø> (ø)`
aicsimageio/readers/ome_tiff_reader.py	`92.98% <63.63%> (-6.11%)`	:arrow_down:
aicsimageio/metadata/utils.py	`90.24% <66.66%> (-0.72%)`	:arrow_down:
aicsimageio/readers/bioformats_reader.py	`84.08% <67.74%> (-1.39%)`	:arrow_down:
aicsimageio/image_container.py	`71.92% <71.92%> (ø)`
aicsimageio/readers/reader.py	`92.52% <73.33%> (-2.63%)`	:arrow_down:
aicsimageio/aics_image.py	`88.31% <73.68%> (+1.61%)`	:arrow_up:
...ts/readers/extra_readers/test_bioformats_reader.py	`96.72% <84.61%> (ø)`
aicsimageio/writers/bfio_writers.py	`85.10% <85.10%> (ø)`
aicsimageio/readers/array_like_reader.py	`97.60% <85.71%> (-2.41%)`	:arrow_down:
... and 35 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Apr 11 '22 17:04 codecov-commenter

Benchmarks

It seems like the OmeTiledTiffWriter performs best on either images larger than 256x256 pixels, and images with a large number of planes. Included in these graphs are the OmeZarrWriter, but it isn't quite ready for a PR. We are still updating bfio to be compliant with v0,4 of NGFF.

Loading 550 images, 16-bit, 2048x2048 pixels, single plane

Small

Loading 15 images, 16-bit ~30,000x50,000 pixels, single plane

Large

Saving data from the AICS test data, wide range of data shapes

Range

Apr 11 '22 18:04 Nicholas-Schaub

Wow, this is really impressive from a quick look. I have a paper deadline this week so unfortunately I will need to hold off on a thorough review until next week.

Apr 12 '22 17:04 evamaxfield

Is there an ETA on this?

May 12 '22 16:05 Nicholas-Schaub

Thanks for the reminder! I will look at this today. So sorry!!

May 12 '22 16:05 evamaxfield

@Nicholas-Schaub to resolve the bfio dependency stuff I had to make a decent amount of admin changes to the repo. Merging main may be a pain and I apologize in advance.

The major changes after merge will be:

whatever tests you have written for this writer should be moved to aicsimageio/tests/writers/extra_writers
you should update the tox.ini config for bfio to run writer tests as well

May 22 '22 02:05 evamaxfield

Check out this pull request on