Keras Generator with RGB images

[AmandaBoatswain]

Hello, I'm trying to create a keras generator using the Augmentor library but I'm running into an issue with the Pil library. My training data is in a numpy array of form [num_samples, width, height, num_channels] where num_channels =3. I tried running the jupyter notebook Augmentor_Keras_Array_Data.ipynb using the mnist dataset and it worked fine, but with the 3 layer images once I try to visualize a sample using the X, y = next(generator) function I get the following errors:

File "C:/Users/amand/Documents/Masters Thesis/Cow Injury Project/cow_injury_CNN/augment_dataset.py", line 61, in X, y = next(generator)

File "C:\Anaconda3\envs\ConvNeuralNet\lib\site-packages\Augmentor\Pipeline.py", line 489, in keras_generator_from_array numpy_array = self._execute_with_array(np.reshape(images[random_image_index], (w, h, l)))

File "C:\Anaconda3\envs\ConvNeuralNet\lib\site-packages\Augmentor\Pipeline.py", line 226, in _execute_with_array pil_image = [Image.fromarray(image)]

File "C:\Anaconda3\envs\ConvNeuralNet\lib\site-packages\PIL\Image.py", line 2426, in fromarray raise TypeError("Cannot handle this data type")

TypeError: Cannot handle this data type

Is this function designed to work with RGB images?

[mdbloice]

Hi, Yeah PIL is very specific about the types of array data it can read. A better option is to use Matplotlib's imshow function:

Import it, and instruct it to display images inline in your notebook:

import matplotlib.pyplot as plt
%matplotlib inline

Then with your array data you can preview it using imshow in some way similar to this:

plt.imshow(X[0].reshape(width, height, 3))

You may or may not need to use reshape to change the array's shape, that depends. The width and height variables above should correspond to the width and height of the image contain in X[0]. Hope this helps. M.

[mdbloice]

By the way, I have updated the example notebooks to use imshow now instead of relying on Image.fromarray.

[AmandaBoatswain]

Thank you for you updates mdbloice!

I actually still ran into the issue while running my code. The error wasn't really in the visualization of the image with matplotlib, it was with the PIL library in the Pipeline.py file when it converts the array to an image file (line 226). In the [Image.fromarray(obj, mode)] function I set the "mode" parameter to RGB, so now it expects a three channel image. There is more info on the accepted modes for this function here: https://pillow.readthedocs.io/en/3.1.x/handbook/concepts.html#concept-modes

[mdbloice]

OK I see, so there is something about the array data what PIL does not like. Well what operations are you actually using in the pipeline? Do you know which operation causes the error?

To me that error sounds like you have array data as floats between 0 and 1 to represent the pixel values, and I think PIL only works with unsigned integers from 0-255.

Are you using the keras_generator() or the keras_generator_from_array() function?

[AmandaBoatswain]

Hello mdbloice,

Here are the specific operations that I used with the library: generator.rotate(probability=1, max_left_rotation=5, max_right_rotation=5) generator.flip_left_right(probability=0.5) generator.random_distortion(probability=1, grid_width=4, grid_height=4, magnitude=8) But like I said previously, the error was fixed when I changed mode parameter in the Image.fromarray() function. I also normalized the images before passing them to the generator and did not encounter any problems with the function.

I am using the keras_generator_from_array() function.

[dipanjanS]

This is confirmed, keras_generator_from_array() is not working on RGB images due to this

C:\Program Files\Anaconda3\lib\site-packages\Augmentor\Pipeline.py in _execute_with_array(self, image)
    226         """
    227 
--> 228         pil_image = [Image.fromarray(image)]
    229 
    230         for operation in self.operations:

It expects a 2D image but we need a 3-D image when using ones with RGB channels

[msayhan]

Hi,

I am working with medical images with 3 channels: RGB. I created a pipeline using the keras_generator_from_array method as follows:

aug_p = Augmentor.Pipeline()
aug_p.shear(probability=1., max_shear_left=25, max_shear_right=25)
aug_p.skew(probability=1., magnitude=1.)
aug_p.random_distortion(probability=1., grid_width=10, grid_height=10, magnitude=10)
y_1hot = Augmentor.Pipeline.categorical_labels(y)
g = aug_p.keras_generator_from_array(X, y_1hot, batch_size=16)
X_hat, y_1hot_hat = next(g)

So, the shape of an input to the generator is (l,x,y,3). However, it leads to the following error message:

KeyError Traceback (most recent call last) /usr/local/lib/python3.5/dist-packages/PIL/Image.py in fromarray(obj, mode) 2459 typekey = (1, 1) + shape[2:], arr['typestr'] -> 2460 mode, rawmode = _fromarray_typemap[typekey] 2461 except KeyError:

KeyError: ((1, 1, 3), '<f4') ... TypeError: Cannot handle this data type

It seems like PIL does not like the format of its input. I can get around this error by converting my images into single-channel, grayscale images and dropping the last dimension. Then, the dimensions become (l,x,y) and the pipeline works. It generates an out of dimensions (l,x,y,1). On the downside, the images loose the color information due to the reduction of channels. I looked into Pipeline.py and tried to understand how things are handled. Basically, it goes like this:

for i in range(batch_size):
     random_image_index = random.randint(0, len(images)-1)
     if np.ndim(images) == 3:  # Grayscale images trigger this branch and all seems to work fine.
         l = 1
     else:                     # RGB images lead into this branch.
         l = np.shape(images)[-1]
     w = images[random_image_index].shape[0]
     h = images[random_image_index].shape[1]
     if l == 1:  # With grayscale images, all seems to work fine.
         numpy_array = self._execute_with_array(np.reshape(images[random_image_index], (w, h)))
     else:       # With RGB images, things break...
         numpy_array = self._execute_with_array(np.reshape(images[random_image_index], (w, h, l)))
     if image_data_format == "channels_first":
        numpy_array = numpy_array.reshape(l, w, h)
     elif image_data_format == "channels_last":
        numpy_array = numpy_array.reshape(w, h, l)
     X.append(numpy_array)
     y.append(labels[random_image_index])
X = np.asarray(X)
y = np.asarray(y)

Also, in _execute_with_array(self, image), there is this line of code: pil_image = [Image.fromarray(image)] Maybe, the error has something to do with the mode of image (https://pillow.readthedocs.io/en/3.1.x/handbook/concepts.html#concept-modes).

Could anyone with a better understanding of Augmentor and how PIL operates come up with a quick fix to the problem? Maybe, setting the mode of image when calling PIL. An update: Actually, I tried setting mode='RGB'. It did not complain about the dimensions, but, images got corrupted so badly. OK. Solved. After setting the mode explicitly to RGB on line 286 as pil_image = [Image.fromarray(image, mode='RGB')], I converted my input images to 8-bit representation via numpy.uint8(). Now, it works like a charm!

By the way, thanks for Augmentor!

[bthorsted]

I have a similar issue, where I have normalized my 3-channel false-color images to floating point with zero mean and standard deviation of 1. However, PIL is really creating tension for me here, since I cannot both use RGB and floating point values. Is the need for PIL deep-seeded in Augmentor or would it be possible to rewrite it without PIL? EDIT: Have you considered using scikit-image rather than PIL? It seems this package has extensive support for all image transforms regardless of the number of channels or the datatype. Just a thought 😄