FileNotFoundError: No such file while fitting

[Diyago]

I'm tried two different datasets from kaggle salt identification challenge (1 grey scale input image) and other one with 3 rgb channels, always get same error

/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/generic_config.py:345: DeprecationWarning: Function Scale() is deprecated. Use Resize instead. Resize has the exactly same interface as Scale. transforms.append(imgaug.augmenters.Scale({"height": self.shape[0], "width": self.shape[1]})) /home/dex/anaconda3/lib/python3.6/site-packages/musket_core/generic_config.py:379: DeprecationWarning: Function Scale() is deprecated. Use Resize instead. Resize has the exactly same interface as Scale. transforms.append(imgaug.augmenters.Scale({"height": self.shape[0], "width": self.shape[1]})) Traceback (most recent call last): File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 75, in load id, x, y = self.proceed(i) File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 93, in proceed item = self.dataset[self.indeces[i]] File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 332, in getitem out = imageio.imread(os.path.join(self.mask, self.ids[item] + "." + self.out_ext)) File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/functions.py", line 206, in imread reader = read(uri, format, 'i', **kwargs) File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/functions.py", line 117, in get_reader request = Request(uri, 'r' + mode, **kwargs) File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/request.py", line 128, in init self._parse_uri(uri) File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/request.py", line 271, in _parse_uri raise FileNotFoundError("No such file: '%s'" % fn) FileNotFoundError: No such file: '/home/dex/Desktop/ml/buildingsegmentation/data_objects/segmentation/bike_2007_005878.png' Traceback (most recent call last): File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 75, in load id, x, y = self.proceed(i) File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 93, in proceed item = self.dataset[self.indeces[i]] File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 332, in getitem out = imageio.imread(os.path.join(self.mask, self.ids[item] + "." + self.out_ext)) File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/functions.py", line 206, in imread reader = read(uri, format, 'i', **kwargs) File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/functions.py", line 117, in get_reader request = Request(uri, 'r' + mode, **kwargs)

Config file:

backbone: resnet18 architecture: Unet augmentation: Fliplr: 0.5

classes: 1 activation: sigmoid #encoder_weights: pascal_voc shape: [128, 128, 3] optimizer: Adam batch: 16 metrics:

• binary_accuracy primary_metric: val_binary_accuracy callbacks: EarlyStopping: patience: 15 monitor: val_binary_accuracy verbose: 1 ReduceLROnPlateau: patience: 4 factor: 0.5 monitor: val_binary_accuracy mode: auto cooldown: 5 verbose: 1 loss: binary_crossentropy stages:
• epochs: 2 negatives: real

[petrochenko-pavel-a]

Hi Dyaugo, according to what I see in the log the problem is concentated around:

File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 332, in getitem
out = imageio.imread(os.path.join(self.mask, self.ids[item] + "." + self.out_ext))

And the problem is that this file, which is probably an image file is not found: /home/dex/Desktop/ml/buildingsegmentation/data_objects/segmentation/bike_2007_005878.png

And this is very strange regarding to what you are sayng here:

I'm tried two different datasets from kaggle salt identification challenge (1 grey scale input image) and other one with 3 rgb channels, always get same error

So I am a little bit confused, may you please show me how you are configuring your dataset?

Thanks in advance, Pavel

[Diyago]

Sure, let me provide full information:

from segmentation_pipeline.impl.datasets import SimplePNGMaskDataSet
from segmentation_pipeline import  segmentation
ds=SimplePNGMaskDataSet("./data_objects/images","./data_objects/segmentation")
cfg = segmentation.parse("people.yaml")
cfg.fit(ds, foldsToExecute = [0])

Error with new dataset:

/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/generic_config.py:345: DeprecationWarning: Function `Scale()` is deprecated. Use `Resize` instead. Resize has the exactly same interface as Scale.
  transforms.append(imgaug.augmenters.Scale({"height": self.shape[0], "width": self.shape[1]}))
/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/generic_config.py:379: DeprecationWarning: Function `Scale()` is deprecated. Use `Resize` instead. Resize has the exactly same interface as Scale.
  transforms.append(imgaug.augmenters.Scale({"height": self.shape[0], "width": self.shape[1]}))
Traceback (most recent call last):
  File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 75, in load
    id, x, y = self.proceed(i)
  File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 93, in proceed
    item = self.dataset[self.indeces[i]]
  File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 332, in __getitem__
    out = imageio.imread(os.path.join(self.mask, self.ids[item] + "." + self.out_ext))
  File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/functions.py", line 206, in imread
    reader = read(uri, format, 'i', **kwargs)
  File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/functions.py", line 117, in get_reader
    request = Request(uri, 'r' + mode, **kwargs)
  File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/request.py", line 128, in __init__
    self._parse_uri(uri)
  File "/home/dex/anaconda3/lib/python3.6/site-packages/imageio/core/request.py", line 271, in _parse_uri
    raise FileNotFoundError("No such file: '%s'" % fn)
FileNotFoundError: No such file: '/home/dex/Desktop/ml/buildingsegmentation/data_objects/segmentation/bike_2007_005878.png'
Traceback (most recent call last):
  File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 75, in load
    id, x, y = self.proceed(i)
  File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 93, in proceed
    item = self.dataset[self.indeces[i]]
  File "/home/dex/anaconda3/lib/python3.6/site-packages/musket_core/datasets.py", line 332, in __getitem__
    out = imageio.imread(os.path.join(self.mas

Dataset is pretty small, just for checking pipeline https://drive.google.com/open?id=1xGUc2KXILpHL4ZTTpSYp05_MOhmWKT3r

[petrochenko-pavel-a]

Hi, I have tried to reproduce your issue and after few minor tweaks it all worked fine for me, my best guess is that your using some old version of SimplePngDataset that does not understand .jpg extension.

backbone: resnet18
architecture: Unet
augmentation:
   Fliplr: 0.5
classes: 1
activation: sigmoid
#encoder_weights: pascal_voc
shape: [224, 224, 3] #- it is hard to guess what sizes work on lower image sizes
optimizer: Adam
batch: 4 #- our epoch should contain at least one batch.
metrics: [binary_accuracy]
primary_metric: val_binary_accuracy
callbacks:
   EarlyStopping:
      patience: 15
      monitor: val_binary_accuracy
      verbose: 1
   ReduceLROnPlateau:
      patience: 4  
      factor: 0.5
      monitor: val_binary_accuracy
      mode: auto
      cooldown: 5
      verbose: 1
loss: binary_crossentropy
stages:
   - epochs: 17
     negatives: reall

Adding start of SimplePNGMaskDataSet to be sure:

class SimplePNGMaskDataSet:
    def __init__(self, path, mask, detect_exts=False, in_ext="jpg", out_ext="png", generate=False):
        self.path = path;
        self.mask = mask;

        ldir = os.listdir(path)

        if ".DS_Store" in ldir:
            ldir.remove(".DS_Store")

        self.ids = [x[0:x.index('.')] for x in ldir]

        self.exts = []

        if detect_exts:
            self.exts = [x[x.index('.') + 1:] for x in ldir]

        self.detect_exts = detect_exts

        self.in_ext = in_ext
        self.out_ext = out_ext

        self.generate = generate

        pass

I can not publish to the pip right now (very long refactoring harms musket core/segmentation pipeline combination stability, hopfully I will finish it and update versions in the nearest couple of days).

For now my advice is - copy paste simple png data set from musket core:

https://github.com/petrochenko-pavel-a/musket_core/blob/master/musket_core/datasets.py#L402

Regards, Pavel