Exception when creating dataset from tiff files

[frcroth]

Context

• Affected library: webknossos python library

I was trying to create a segmentation dataset from tiff images. For this, I created grayscale png files and converted them to tiff with imagemagick. Then I created the dataset with this script:

from pathlib import Path

from webknossos import Dataset
from webknossos.dataset import SEGMENTATION_CATEGORY

def main():
    INPUT_DIR = Path("/home/felix/Desktop/test-basic-dataset/tiffs")
    OUTPUT_DIR = Path("/home/felix/Desktop/test-basic-dataset/output")

    dataset = Dataset.from_images(
        input_path=INPUT_DIR,
        output_path=OUTPUT_DIR,
        voxel_size = (1,1,1),
        layer_category=SEGMENTATION_CATEGORY)

if __name__ == "__main__":
    main()

With these files: tiffs.zip When running this script, I am getting the following output:

/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/pims/api.py:204: UserWarning: <class 'webknossos.dataset._utils.pims_imagej_tiff_reader.PimsImagejTiffReader'> errored: /home/name/Desktop/test-basic-dataset/tiffs/62.tiff is not an ImageJ Tiff
  warn(message)
/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/pims/api.py:204: UserWarning: <class 'webknossos.dataset._utils.pims_imagej_tiff_reader.PimsImagejTiffReader'> errored: /home/name/Desktop/test-basic-dataset/tiffs/62.tiff is not an ImageJ Tiff
  warn(message)
Creating layer from images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 0:00:00 | -:--:--INFO:root:Writing chunk BoundingBox(topleft=(0, 0, 0), size=(64, 64, 32))
INFO:root:Writing chunk BoundingBox(topleft=(0, 0, 0), size=(64, 64, 32))
Creating layer from images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 | 0:00:00
Traceback (most recent call last):
  File "/home/name/Desktop/test-basic-dataset/create.py", line 44, in <module>
    main()
  File "/home/name/Desktop/test-basic-dataset/create.py", line 11, in main
    dataset = Dataset.from_images(
  File "/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/webknossos/dataset/dataset.py", line 635, in from_images
    ds.add_layer_from_images(
  File "/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/webknossos/dataset/dataset.py", line 1258, in add_layer_from_images
    executor.map_to_futures(func_per_chunk, args),
  File "/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/cluster_tools/executors/multiprocessing_.py", line 264, in map_to_futures
    futs = [self.submit(fn, arg) for arg in args]
  File "/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/cluster_tools/executors/multiprocessing_.py", line 264, in <listcomp>
    futs = [self.submit(fn, arg) for arg in args]
  File "/home/name/Desktop/test-basic-dataset/env/lib/python3.10/site-packages/cluster_tools/executors/multiprocessing_.py", line 147, in submit
    fut = submit_fn(__fn, *args, **kwargs)
  File "/usr/lib/python3.10/concurrent/futures/process.py", line 722, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

So, there is an exception. Also, it randomly states that one of the tiff files is invalid (which is weird because there are only 2 unique tiff files, since everything else is copied from them). From what the dataset looks like, it mostly worked. There is one part of the image that has no segment id (0), not sure if that is related.

Your Environment for bug

• Operating System and version: Linux 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
• Version of webKnossos-libs (Release or Commit): 0.14.15