FileNotFound Error while training a new model

[ThaoCao]

Hi all, I am wondering if anyone has encountered this error: FileNotFoundError: The directory '/home/kcutler/DataDrive/debug' does not exist.

I am trying to train a new omnipose model (linux terminal - GPU). In omnipose venv, the command is: python -m omnipose --train --use_gpu --dir /home/thaocao/omnipose_training_images1/ --img_filter _img --mask_filter _masks --n_epochs 4000 --pretrained_model bact_phase_omni_2 --learning_rate 0.1 --diameter 0 --batch_size 16 --RAdam --all_channels --nclasses 2 --tyx 128,128 --save_each I see that a folder 'models' was created within my given directory of training images, but nothing was saved. The images are single channel .tif in 8-bit. The masks are binary .tif in 8-bit with foreground =1 and background=0.

If anyone has any thoughts, I'd be hugely grateful! Thanks

[Jolanda5]

I got the same error when trying to train. The problem comes from omnipose/core.py, line 2360, it looks like there is a debug save not commented out. If you have the code you can comment that line out. Although then, there will be a ValueError raised because the data didn't look as omnipose expected it to.

2023-09-28 20:17:56,101 [CRITICAL] Sparse or over-dense image detected. Problematic index is: 4636. Image shape is: (2, 300, 300). tyx is: (224, 224). rescale is 1.0 2023-09-28 20:17:57,345 [CRITICAL] Sparse or over-dense image detected. Problematic index is: 538. Image shape is: (2, 520, 696). tyx is: (224, 224). rescale is 1.0 Original Traceback (most recent call last): File "lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop data = fetcher.fetch(index) File "lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 51, in data = [self.dataset[idx] for idx in possibly_batched_index] File "lib/python3.10/site-packages/omnipose/data.py", line 292, in getitem imgi[i], labels[i], scale[i] = random_crop_warp(img=self.data[idx], File "lib/python3.10/site-packages/omnipose/core.py", line 2422, in random_crop_warp return random_crop_warp(img, Y, tyx, v1, v2, nchan, rescale, scale_range, File "lib/python3.10/site-packages/omnipose/core.py", line 2422, in random_crop_warp return random_crop_warp(img, Y, tyx, v1, v2, nchan, rescale, scale_range, File "lib/python3.10/site-packages/omnipose/core.py", line 2422, in random_crop_warp return random_crop_warp(img, Y, tyx, v1, v2, nchan, rescale, scale_range, [Previous line repeated 98 more times] File "lib/python3.10/site-packages/omnipose/core.py", line 2360, in random_crop_warp skimage.io.imsave('/home/kcutler/DataDrive/debug/img'+str(depth)+'.png',img[0])** File "lib/python3.10/site-packages/skimage/io/_io.py", line 143, in imsave return call_plugin('imsave', fname, arr, plugin=plugin, **plugin_args) File "lib/python3.10/site-packages/skimage/io/manage_plugins.py", line 207, in call_plugin return func(*args, **kwargs) File "lib/python3.10/site-packages/imageio/v2.py", line 263, in imwrite with imopen(uri, "wi", **imopen_args) as file: File "lib/python3.10/site-packages/imageio/core/imopen.py", line 113, in imopen request = Request(uri, io_mode, format_hint=format_hint, extension=extension) File "lib/python3.10/site-packages/imageio/core/request.py", line 247, in init self._parse_uri(uri) File "lib/python3.10/site-packages/imageio/core/request.py", line 412, in _parse_uri raise FileNotFoundError("The directory %r does not exist" % dn) FileNotFoundError: The directory '/home/kcutler/DataDrive/debug' does not exist

[kevinjohncutler]

Sorry I didn't see this. @Jolanda5 @ThaoCao, this error comes up if your images are too sparse or completely full of cell pixels, indicating that your training data is not well-formatted. I should replace this with an alternate directory or just kill the process, but I suppose this accidentally accomplishes the latter. In the meantime, you can edit this save command so that you can see which image is giving you the error.

[Jolanda5]

My solution was to comment the data checking and recursion out because we want our training to include negative GT (GT where the network is not supposed to detect anything).