Mask_RCNN
Mask_RCNN copied to clipboard
matterport
Errors when using imgaug's augmentation.
Hello, I'm getting this error each time I'm trying to use augmentation.
SuspiciousSingleImageShapeWarning: You provided a numpy array of shape (1024, 1024, 37) as a single-image augmentation input, which was interpreted as (H, W, C). The last dimension however has a size of >=32, which indicates that you provided a multi-image array with shape (N, H, W) instead. If that is the case, you should use e.g. augmenter(imageS=
) or augment_imageS( ). Otherwise your multi-image input will be interpreted as a single image during augmentation.
My config and the code:
IMAGES_PER_GPU = 1
NUM_CLASSES = 1 + 1
STEPS_PER_EPOCH = 500
DETECTION_MIN_CONFIDENCE = 0.7
RPN_ANCHOR_SCALES = (16, 32, 64, 128, 256)
MINI_MASK_SHAPE = (112, 112)
max_augs = 3
augmentation = iaa.SomeOf((0, max_augs), [
iaa.Fliplr(0.5),
iaa.Flipud(0.5),
iaa.GaussianBlur(sigma=(0.0, 5.0))
])
...
model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=100,
augmentation=augmentation,
layers='heads'
)
What am I doing wrong? Thank you in advance.
As the warning says, il will interprete this as one single image of size 10241024 with 32 channels instead of 32 images of size 10241024 (1channel). Because I admit 1024102432 are masks for one image right ? Do you get an error with this ? Or is it just a warning. Anyway, in model.py/load_image_gt, this is where you call augment_image(), just replace this by augment_images(). This is how I understand this warning
As the warning says, il will interprete this as one single image of size 1024_1024 with 32 channels instead of 32 images of size 1024_1024 (1channel). Because I admit 1024_1024_32 are masks for one image right ? Do you get an error with this ? Or is it just a warning. Anyway, in model.py/load_image_gt, this is where you call augment_image(), just replace this by augment_images(). This is how I understand this warning
Thank you. I'm just getting a warning and it seems it doesn't affect the training, but I'm not sure the augmentation actually works in this case. Will try with augment_imageS().
I got the same warning when I am training on the nucleus dataset.
Is there anyone can explain how does this happend or maybe why and how to fix it?
I find a docs :
https://imgaug.readthedocs.io/en/latest/_modules/imgaug/augmenters/base.html
The problem is in mrcnn/model.py, load_image_gt function.
Take a look at this code:
image = dataset.load_image(image_id)
mask, class_ids = dataset.load_mask(image_id)
...
if augmentation:
...
image = det.augment_image(image) # 1
mask = det.augment_image(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook)) # 2
Let's imagine you've loaded an image and its image.shape == (1024, 1024, 3). A typical RGB image.
Then let mask.shape be equal to (1024, 1024, 37). If so, class_ids is of shape (37,). What does it mean? It means that there are 37 objects on the image and each got its own binary mask.
Let's move further. The warning are generated by imgaugs augment_image function.
# 1 never throws the warning because 3 channels in an image its fine. It shouts when you call it with the mask because there are too many channels for a normal image. All you have to do is just replace # 2 with this line:
mask = det.augment_images(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook))
The only change is using of kind of plural variant of augment_image.
One suggestion: if the number of mask in your pictures may varying from 0 to many. You can slightly modify the load_image_gt function in mrcnn/model.py as following:
if mask_shape[-1] == 1:
mask = det.augment_image(mask.astype(np.uint8),
hooks=imgaug.HooksImages(activator=hook))
else:
mask = det.augment_images(mask.astype(np.uint8),
hooks=imgaug.HooksImages(activator=hook))
This fixed my issue. Thank you.
mask = det.augment_images(mask.astype(np.uint8),
hooks=imgaug.HooksImages(activator=hook))
This does not solve the issue: I changed my code like @ZhendongYuan suggested. As a result the warning is gone, but the masks are not correctly augmented.
Correct masks
mask = det.augment_image(mask.astype(np.uint8),
hooks=imgaug.HooksImages(activator=hook))
Incorrect masks
mask = det.augment_images(mask.astype(np.uint8),
hooks=imgaug.HooksImages(activator=hook))
@johanwillemsen1991 Me too,my training failed. I changed my code image to images.
@3goto Did you find any solution? I changed my code image to images.
Till now, I am facing the same problem. The training process failed after some iterations.
@hafizurr I just continued with image, and for me the training process went okay, as I got a reasonable model out of it.
@creator-947 At first, I used more than 8 augmentation sequences. Then, I reduced it, and worked for me. Perhaps, my computer GPU couldn't run the training process for a higher number of sequences. My suggestion is that try a lower number of augmentation sequences. If it works then increase your augmentation sequence.
N.B.: Augmentation sequence means crop operation, horizontal flips operation, gaussian blur operations, etc.
Is this really an issue? Without looking deeper in the code, wouldn't we want ALL masks in an image be augmented the same way? If the image itself is augmented eg flipped, all masks would have to be augmented/flipped as well to match it, or not?
As @johanwillemsen1991, I can also confirm using det.augment_images (in plural) does not work. To see that, you can use the inspect_data notebook, go to the image augmentation cell and pass some kind of augmentation, such as:
import imgaug
# Add augmentation and mask resizing.
augmentations = imgaug.augmenters.Sometimes(1,
imgaug.augmenters.OneOf
(
[
imgaug.augmenters.Fliplr(1),
imgaug.augmenters.Flipud(1),
]
)
)
image, image_meta, class_ids, bbox, mask = modellib.load_image_gt(
dataset, config, image_id, augmentation=augmentations, use_mini_mask=True)
log("mask", mask)
display_images([image]+[mask[:,:,i] for i in range(min(mask.shape[-1], 7))])
And you'll see that augment_images does not augment the masks properly.
faced it in
imgaug 0.4.0. and usingimgaug 0.2.6was good to go.
thank you
One suggestion: if the number of mask in your pictures may varying from 0 to many. You can slightly modify the load_image_gt function in mrcnn/model.py as following:
if mask_shape[-1] == 1: mask = det.augment_image(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook)) else: mask = det.augment_images(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook))
As @ZhendongYuan said this method is right. All you need to do is add an extra clause to that if statement and make it
if mask_shape[-1] == 1 or mask_shape[-1] == 3.. Then this works because both 1 and 3 channel images are considered as single images.
I tried it out and so far it is working for me. Will report if this gives any future errors
One suggestion: if the number of mask in your pictures may varying from 0 to many. You can slightly modify the load_image_gt function in mrcnn/model.py as following:
if mask_shape[-1] == 1: mask = det.augment_image(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook)) else: mask = det.augment_images(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook))As @ZhendongYuan said this method is right. All you need to do is add an extra clause to that if statement and make it
if mask_shape[-1] == 1 or mask_shape[-1] == 3.. Then this works because both 1 and 3 channel images are considered as single images.I tried it out and so far it is working for me. Will report if this gives any future errors
@ankit16-py and @ZhendongYuan I am still getting the same warning even after adding the if clause
One suggestion: if the number of mask in your pictures may varying from 0 to many. You can slightly modify the load_image_gt function in mrcnn/model.py as following:
if mask_shape[-1] == 1: mask = det.augment_image(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook)) else: mask = det.augment_images(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook))As @ZhendongYuan said this method is right. All you need to do is add an extra clause to that if statement and make it
if mask_shape[-1] == 1 or mask_shape[-1] == 3.. Then this works because both 1 and 3 channel images are considered as single images.I tried it out and so far it is working for me. Will report if this gives any future errors
I have tried replacing det.augment_image by det.augment_images as described and this does not work. The warning disappears but the validation loss skyrockets and the predictions made by the resulting model are complete nonsense.
det.augment_images expects a multi-image of shape (N, H, W). The masks are of shape (H, W, N). For many augmentation methods, everything works well for the masks when using det.augment_image. det.augment_images cannot work here.
The solution is to ignore the warning and double-check if the augmentation method works for the masks.
One suggestion: if the number of mask in your pictures may varying from 0 to many. You can slightly modify the load_image_gt function in mrcnn/model.py as following:
if mask_shape[-1] == 1: mask = det.augment_image(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook)) else: mask = det.augment_images(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook))As @ZhendongYuan said this method is right. All you need to do is add an extra clause to that if statement and make it
if mask_shape[-1] == 1 or mask_shape[-1] == 3.. Then this works because both 1 and 3 channel images are considered as single images. I tried it out and so far it is working for me. Will report if this gives any future errorsI have tried replacing det.augment_image by det.augment_images as described and this does not work. The warning disappears but the validation loss skyrockets and the predictions made by the resulting model are complete nonsense.
det.augment_images expects a multi-image of shape (N, H, W). The masks are of shape (H, W, N). For many augmentation methods, everything works well for the masks when using det.augment_image. det.augment_images cannot work here.
The solution is to ignore the warning and double-check if the augmentation method works for the masks.
Yes this is true for me to. So be careful while changing from augment_image to augment_images and keep an eye on that validation loss.
One suggestion: if the number of mask in your pictures may varying from 0 to many. You can slightly modify the load_image_gt function in mrcnn/model.py as following:
if mask_shape[-1] == 1: mask = det.augment_image(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook)) else: mask = det.augment_images(mask.astype(np.uint8), hooks=imgaug.HooksImages(activator=hook))As @ZhendongYuan said this method is right. All you need to do is add an extra clause to that if statement and make it
if mask_shape[-1] == 1 or mask_shape[-1] == 3.. Then this works because both 1 and 3 channel images are considered as single images.I tried it out and so far it is working for me. Will report if this gives any future errors
As mentioned previously, I am reporting my findings with the intention that it may help people who are in similar situations.
-
ALWAYS USE AUGMENTATION- regardless of how big your data is, I noticed that when you train the entire model (heads and everything) it tends to overfit after a few epochs due to the sheer size of the model. This problem is solved by using some augmentation. For my particular problem the difference in the results are staggering.
-
USE THE "det.augment_images" WISELY- as mentioned by @OBrink the input dimensions are different for "augment_image" and "augment_images". I am pretty sure that most of us are using dimensions as needed by "augment_image" and you will run into an error when using "augment_images" because we are just not loading the masks in that manner.
-
CHECK IF "det.augment_image" IS ACTUALLY WORKING FOR YOUR CASE- Train the model for a bit and see if the "det.augment_image" is actually working (see if your val_losses are decreasing overtime). If it does work you can safely ignore the warning presented by imgaug while training. I understand that it becomes a pain to deal with huge number of warnings which cause you to tediously scroll the logs in the terminal to see the per epoch outputs. To combat that you can simply write the outputs (loss values) to a json file after each epoch and view the results from there (use custom callbacks).
In order to facilitate verifying if an image augmentation method does what it is supposed to do, I added the section Image augmentation to the notebook inspect_balloon_data.ipynb in my fork of repository. Here, you can simply choose the augmentation method that you want to apply and a random image from the balloon dataset with the bounding boxes and masks is visualized before and after the application of the augmentation. Here, you can also easily check what it does to your image when you use 'augment_images' instead of 'augment_image'. Have a look here.
@sohinimallick, you have to check that the augmentations are actually applied the right way (check the notebook I mentioned in my last comment to visualize what the augmentations are actually doing to your images). If you have not changed anything significant in the Mask RCNN training code, you can probably ignore the warning.
mask = det.augment_images(mask.astype(np.uint8),hooks=imgaug.HooksImages(activator=hook)) masks me loss value start with a large number
我找到一个文档：
https://imgaug.readthedocs.io/zh-CN/latest/_modules/imgaug/augmenters/base.html
how to use it? thanks
In order to facilitate verifying if an image augmentation method does what it is supposed to do, I added the section Image augmentation to the notebook inspect_balloon_data.ipynb in my fork of repository. Here, you can simply choose the augmentation method that you want to apply and a random image from the balloon dataset with the bounding boxes and masks is visualized before and after the application of the augmentation. Here, you can also easily check what it does to your image when you use 'augment_images' instead of 'augment_image'. Have a look here.
@OBrink Thanks a lot for sharing the notebook. Looks like many useful transformations like Cutout, pixel dropout, Add cannot be used. Did you, by any chance, venture into finding out how these augmentations can be incorporated as well?
In order to facilitate verifying if an image augmentation method does what it is supposed to do, I added the section Image augmentation to the notebook inspect_balloon_data.ipynb in my fork of repository. Here, you can simply choose the augmentation method that you want to apply and a random image from the balloon dataset with the bounding boxes and masks is visualized before and after the application of the augmentation. Here, you can also easily check what it does to your image when you use 'augment_images' instead of 'augment_image'. Have a look here.
@OBrink Thanks a lot for sharing the notebook. Looks like many useful transformations like Cutout, pixel dropout, Add cannot be used. Did you, by any chance, venture into finding out how these augmentations can be incorporated as well?
@AshwinAKannan Sorry, when working with Mask R CNN, I have only ever worked with the augmentations that did work without further modifications. You might have to write custom functions that apply the augmentations in order to not to destroy the masks. For example, in the case of pixel dropout, it should be enough to only apply the augmentation to the image while the mask remains unchanged. It would definitely be possible but it would mean extra work.