Extracting region proposals

[djongpo]

Really a great work. Helped me to understand better. Can you please tell me the steps to follow to get region proposals for a test image?

[praisethemoon]

Hello, I feel your pain. I believe the regions are in the following file and line: https://github.com/kentaroy47/frcnn-from-scratch-with-keras/blob/master/test_frcnn.py#L170

where:

	# convert from (x1,y1,x2,y2) to (x,y,w,h)
	R[:, 2] -= R[:, 0]
	R[:, 3] -= R[:, 1]

I think these are the raw region proposals relative to the feature map size (not the image image) thus a transformation is needed. Maybe multiply them by C.rpn_stride*x, C.rpn_stride*y, C.rpn_stride*(x+w), C.rpn_stride*(y+h) as in line 217 of the same file? Let me know how it goes.

[djongpo]

@praisethemoon Actually I have only gone-through the code. I believe the line python train_rpn.py --network mobilenetv1 -o simple -p /path/to/your/dataset/ given in README.md is training the RPN module on my own data set. I have only some images of which I have to extract the region proposals. Could you explain me in detail the commands to run/portions to edit for acheiving the same? As I can figure out that probably you have tested the code yourself.

Thanks in advance. Waiting for a reply.

[GauravKrRoy]

@djongpo I also faced a similar issue while implemeting the paper 'Perceptual GAN for small object detection' where I need to do RoI pooling on a feature map for that I need the proposals. I wonder if you would help me out?

[kentaroy47]

@GauravKrRoy https://github.com/kentaroy47/frcnn-from-scratch-with-keras/blob/master/train_frcnn.py#L212-L214

RPNs and ROIs are extracted in the above lines.

You should take the outputs there and use it in your part of research.

I used VOC2007 for training, FYI https://pjreddie.com/projects/pascal-voc-dataset-mirror/

[GauravKrRoy]

Hey @kentaroy47 I see that you have predicted the region proposals and then found out the region of interest bounding boxes in https://github.com/kentaroy47/frcnn-from-scratch-with-keras/blob/6d173775b94879e9b7cb36f31bc75a0b99b72485/train_rpn.py#L213-L216 . I really appreciate your work of separately training the RPN network. It will be very helpful in my research work if I can use the train_rpn.py file for predicting on my test images. I wonder if these lines(L213-L216) would do my work? I also want to save the model of RPN training, I believe that I should include model_rpn.save() before https://github.com/kentaroy47/frcnn-from-scratch-with-keras/blob/6d173775b94879e9b7cb36f31bc75a0b99b72485/train_rpn.py#L224

[kentaroy47]

@GauravKrRoy yes, you should save the model at the end of training. then, if you load the model and use model.predict, you can get the roi for the input images. I can write the script but will take some time..

[djongpo]

Hey, @kentaroy47 @GauravKrRoy I wrote a separate code, where I load the saved trained model and predict the region proposals from that. Here is the code:

import random
import pprint
import keras
import sys
import time
import numpy as np
from optparse import OptionParser
import pickle

from keras import backend as K
from keras.optimizers import Adam, SGD, RMSprop
from keras.layers import Input
from keras.models import Model
from keras_frcnn import data_generators
from keras_frcnn import config
from keras_frcnn import losses as losses
import keras_frcnn.roi_helpers as roi_helpers
from keras.utils import generic_utils
import cv2
from keras.models import load_model
image = cv2.imread("/home/pg2017/cse/17071016/img_519.jpg")
model = load_model("RPN_model1.h5")
P_rpn = model.predict(image)
R = roi_helpers.rpn_to_roi(P_rpn[0], P_rpn[1], C, K.image_dim_ordering(), use_regr=True, overlap_thresh=0.7, max_boxes=300)
X2, Y1, Y2, IouS = roi_helpers.calc_iou(R, img_data, C, class_mapping)
print(X2, Y1, Y2, IouS)

but it gives error as:

  File "rpn_test.py", line 23, in <module>
    model = load_model("RPN_model1.h5")
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/engine/saving.py", line 312, in _deserialize_model
    sample_weight_mode=sample_weight_mode)
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/engine/training.py", line 137, in compile
    loss_functions = [losses.get(l) for l in loss]
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/engine/training.py", line 137, in <listcomp>
    loss_functions = [losses.get(l) for l in loss]
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/losses.py", line 133, in get
    return deserialize(identifier)
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/losses.py", line 114, in deserialize
    printable_module_name='loss function')
  File "/home/pg2017/cse/17071016/.conda/envs/myenv/lib/python3.7/site-packages/keras/utils/generic_utils.py", line 165, in deserialize_keras_object
    ':' + function_name)
ValueError: Unknown loss function:rpn_loss_cls_fixed_num

I can't figure it out. Can you help @kentaroy47 ? Did you do it @GauravKrRoy ?

[kentaroy47]

@djongpo

when loading custom loss functions, we need to pass that during model loading. https://github.com/keras-team/keras/issues/5916

from keras_frcnn.losses import rpn_loss_cls_fixed_num, rpn_loss_cls
model = load_model("RPN_model1.h5", custom_objects={'rpn_loss_cls_fixed_num': rpn_loss_cls_fixed_num})

should work. just define rpn_loss_cls_fixed_num and pass it to the model.

BTW, I'm not sure why that pops up when loading rpns (didn't happen to me). might be a Keras version issue?

[marvision-ai]

@kentaroy47 Thank you for the great repo! It is a pleasure reading through the code.

I cannot seem to figure out how to extract the region proposals and put bounding boxes around them. The script djongpo wrote does not work on my machine because it cannot load the model for some reason. Also, my model was saved as a .hdf5 instead of .h5?

Have you worked on the script to do test_rpn.py? I would love to see what I am doing wrong in loading the model and then applying it to an image.

thanks @kentaroy47 !

[praveench2398]

@djongpo @kentaroy47 @GauravKrRoy

@djongpo

when loading custom loss functions, we need to pass that during model loading. keras-team/keras#5916

from keras_frcnn.losses import rpn_loss_cls_fixed_num, rpn_loss_cls
model = load_model("RPN_model1.h5", custom_objects={'rpn_loss_cls_fixed_num': rpn_loss_cls_fixed_num})

should work. just define rpn_loss_cls_fixed_num and pass it to the model.

BTW, I'm not sure why that pops up when loading rpns (didn't happen to me). might be a Keras version issue?

when iam predicting region proposals using model.predict it is showing that expected input_1 to have 4 dimensions, but got array with shape (1632, 1280, 3).how can i solve that can you please help me

[linaemunsamy]

@kentaroy47 @marvision-ai @GauravKrRoy can i use the same load_model method for the main frcnn? has anyone tried to load the main model and predict? any help will be greatly appreciated!