ssd_keras
ssd_keras copied to clipboard
rykov8
Trainning very slow
My gpu is Titan X And I use my own dataset to train this net work below is the result
Epoch 1/30 188/2875 [>.............................] - ETA: 3622s - loss: 1.6571
As you can see,every second I just step one and I need about 3000 seconds to go through one epoch? Is there some wrong with me?
Hi,
do you have anyway to check your gpu utilization ? Under ubuntu I use nvidia-smi -q -g 0 -d UTILIZATION -l.
If you're not at a 100% of gpu usage (or the allocated percentage from your code) you may want to check the generator, if i remember well the inputs and targets arrays are increased in size at each iteration. You could try to preallocate to avoid useless work/memory access from your cpu.
Hi @liu6381810 , Sorry I don't have a solution to your problem but I have also been trying to train the network to my dataset and receive the following error. Any help would be greatly appreciated. Thanx!:
FileNotFoundError: [Errno 2] No such file or directory: '../../frames/frame02579.png'
I changed the input path for dataset(images) to try to avoid the error.Below is the change I made:
path_prefix = 'C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/JPEGImages/' gen = Generator(gt, bbox_util, 16, 'C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/JPEGImages/', train_keys, val_keys, (input_shape[0], input_shape[1]), do_crop=False)
But I still receive the following error.It is still trying to find nonexistent 'frame02579' in my folder path: FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/JPEGImages/frame02579.png'
@D3lt4lph4 Hi,thanks for your reply. I have used watch nvidia-smi to see the gpu utilization, and I found that it jump to 100% and then jump to 0%, I guess the reason is the generator need to cost time to generate targets and imgs. I changed nothing except gt and image path, and i use for x,y in gen.generate(True): print x.shape,y.shape
and i got (16,300,300,3) (16,7308,8) so I think the generator yield a batch of 16 every time, but i don't know why each time the 188/2875 [>.............................] - ETA: 3622s - loss: 1.6571 just step one which means 188 change to 189 but not 188+16 = 204.
@liu6381810 , Thanks for the quick reply! I have changed it actually. I ran the following python file 'get_data_from_XML.py' with the following changes: data = XML_preprocessor('C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/Annotations/').data pickle.dump(data,open('VOC2007.p','wb'))
In SSD_training, I replaced 'gt_pascal.pkl' with 'VOC2007.p' in gt = pickle.load(open('gt_pascal.pkl', 'rb')) I also changed NUM_CLASSES to 21.i.e NUM_CLASSES = 21 (Pascal voc has 20 classes)
@ManjeeraJagiri You should check whether gt has "frame02579.png" this key
@liu6381810 'VOC2007.p' doesn't have it. The original pickle file 'gt_pascal.pkl' has it. But when I have loaded gt the following way in the code: gt = pickle.load(open('VOC2007.p', 'rb')). Why is it still searching for "frame02579.png" which was present in 'gt_pascal.pkl' and outputting the error FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/JPEGImages/frame02579.png'?
@liu6381810 if you modified nothing, the batches are of size 16, as for the one step it is suppose to go that way, from what I see, you should have something like 30 epochs, with 2875 batches of size 16 (with 2875 being your number of training images, it is set that way in the example file).
Everything should be define in the Generator(gt, bbox_util, 16,...) and the model.fit_generator(...) from the SSD_training.ipynb.
Basically I had the same problem as yours with my gpu dropping from 100% to 0%. The following is what I modified in the Generator definition, there are two things: first I preallocate the inputs and targets arrays, and second I have removed one .astype('float32'). I'm not sure whether both or just one of the modifications helped since I did both at the same time, but here is the code :
def generate(self, train=True):
while True:
#New iterator needed
i = 0
if train:
shuffle(self.train_keys)
keys = self.train_keys
else:
shuffle(self.val_keys)
keys = self.val_keys
# Preallocation of the arrays
inputs = np.zeros((16,300,300,3), dtype=K.floatx())
targets = np.zeros((16,7308,8), dtype=K.floatx())
for key in keys:
img_path = self.gt.fullPath[key]
#Removing of the astype('float32')
img = imread(img_path)
y = self.gt.y[key].copy()
if train and self.do_crop:
img, y = self.random_sized_crop(img, y)
img = imresize(img, self.image_size).astype('float32')
if train:
shuffle(self.color_jitter)
for jitter in self.color_jitter:
img = jitter(img)
if self.lighting_std:
img = self.lighting(img)
if self.hflip_prob > 0:
img, y = self.horizontal_flip(img, y)
if self.vflip_prob > 0:
img, y = self.vertical_flip(img, y)
y = self.bbox_util.assign_boxes(y)
# Filling the arrays
inputs[i] = img
targets[i] = y
i += 1
if i == self.batch_size:
i=0
yield preprocess_input(inputs), targets
Also you should be careful with the astype('float32') if you try to remove it, I am not completely sure of the implications, it should be fine but not 100% sure you will not lose precision (and in my case I had way bigger images (4000x3000) before the resize that's why I tried to remove it).
EDIT There was a mistake on the last if that I just saw and corrected. But now, the training will go slow again. After testing, the main problem is still on the cpu side, and I have at least 2 bottlenecks, the resizing and the image transformation. The only way I see to prevent them is to pre-resized the images and to directly load to the correct size and maybe to pre-create a set of already transformed images to load.
@D3lt4lph4 Ok, I will try it tomorrow,but why 2875 batches of size 16 ? from the generate() function, we can see that I should have 2875 / 16 batches of size 16
for key in keys:
this for loop should just loop 2875 times, and just when i == 16 it yields a batch of 16 images and targets
@ManjeeraJagiri I think you should print gt["frame02579.png"] to see whether your gt has this key. If not I think it should not search for this image, and you can put your code here so we can know more clearly
@liu6381810 Yes and no, it is true that you generate batches of 16 out of the total number of images in the generate() function, but when you do history = model.fit_generator(gen.generate(True), gen.train_batches,...) the gen.train_batches refers to the number of steps per epoch and if you look in the Generator class you have self.train_batches = len(train_keys), your number of images in the training set.
So in the end you have 30 epochs with as many steps as you have images and each step uses one batch. The generate() function is just here to provide the batches, once you have done the for key in keys: loop you will start it again until you reach the end of your epoch.
@D3lt4lph4 Yes, It solves my problem. And my generator output imgs and targets, the targets' shape is (7308,14) do you know what does 14 mean?
from fucntion "assign_boxes" in ssd_utils.py i see this
Return assignment: Tensor with assigned boxes, numpy tensor of shape (num_boxes, 4 + num_classes + 8), priors in ground truth are fictitious, assignment[:, -8] has 1 if prior should be penalized or in other words is assigned to some ground truth box, assignment[:, -7:] are all 0. See loss for more details.
my num_classes = 2 so 14 is correct , but i don't know what does it mean
@liu6381810 As far as I understand, the targets have the same shape as the predictions. For the prediction, the first 4 cells in the array are the regressed location of your box (check the original paper for more details here), then you have one cell per class (basically to tell the class of your box) and finally the last 8 are for the coordinates of the ground truth box and the variances (see #53 for more details on the variances). For the targets, the first part is the same as for the predictions and it seems that the last 8 are of no use apart from the penalization cell at -8. As to why the lasts are zeros, I don't really know but it possible that keras do not allow targets with a different size than the size of the predictions.
@D3lt4lph4 Sorry to edit above question slowly, I have known what the assignment and encoded box mean. My training data is to detect a very small object (about 1520 pixel in 300300 and only one every img) So every image I just have 1-3 priors correspond to my ground truth box After training, model tends to predict all the priors to be back ground, and even this the loss is very low my val_loss is about 0.1 so is there some method to solve this problem?
@liu6381810 I guess there was some issue with the pickle file. I started freshly and ran the code again. It is training now(I am running in CPU, going to take a long time). Thanks a lot for your help :) Greatly appreciated! If its okay, I actually have another doubt. How do I save this model? Since it is fine tuned, a simple model.save and model.load wont work,( I assume) My code:
import cv2
import keras
from keras.applications.imagenet_utils import preprocess_input
from keras.backend.tensorflow_backend import set_session
from keras.models import Model
from keras.preprocessing import image
import matplotlib.pyplot as plt
import numpy as np
import pickle
from random import shuffle
from scipy.misc import imread
from scipy.misc import imresize
import tensorflow as tf
from ssd import SSD300
from ssd_training import MultiboxLoss
from ssd_utils import BBoxUtility
#%matplotlib inline
plt.rcParams['figure.figsize'] = (8, 8) #set the matplotlib figure default size in ipython notebook?
plt.rcParams['image.interpolation'] = 'nearest' #we’ll set the interpolation to ‘nearest neighborhood’ so that it’s easier to distinguish individual pixels in your image
np.set_printoptions(suppress=True)
# config = tf.ConfigProto()
# config.gpu_options.per_process_gpu_memory_fraction = 0.9
# set_session(tf.Session(config=config))
# some constants
NUM_CLASSES = 21
input_shape = (300, 300, 3)
priors = pickle.load(open('prior_boxes_ssd300.pkl', 'rb'))
bbox_util = BBoxUtility(NUM_CLASSES, priors)
gt = pickle.load(open('VOC2007.p', 'rb'))
keys = sorted(gt.keys())
num_train = int(round(0.8 * len(keys)))
train_keys = keys[:num_train]
val_keys = keys[num_train:]
num_val = len(val_keys)
#print (gt["frame02579.png"])
class Generator(object):
def __init__(self, gt, bbox_util,
batch_size, path_prefix,
train_keys, val_keys, image_size,
saturation_var=0.5,
brightness_var=0.5,
contrast_var=0.5,
lighting_std=0.5,
hflip_prob=0.5,
vflip_prob=0.5,
do_crop=True,
crop_area_range=[0.75, 1.0],
aspect_ratio_range=[3./4., 4./3.]):
self.gt = gt
self.bbox_util = bbox_util
self.batch_size = batch_size
self.path_prefix = path_prefix
self.train_keys = train_keys
self.val_keys = val_keys
self.train_batches = len(train_keys)
self.val_batches = len(val_keys)
self.image_size = image_size
self.color_jitter = []
if saturation_var:
self.saturation_var = saturation_var
self.color_jitter.append(self.saturation)
if brightness_var:
self.brightness_var = brightness_var
self.color_jitter.append(self.brightness)
if contrast_var:
self.contrast_var = contrast_var
self.color_jitter.append(self.contrast)
self.lighting_std = lighting_std
self.hflip_prob = hflip_prob
self.vflip_prob = vflip_prob
self.do_crop = do_crop
self.crop_area_range = crop_area_range
self.aspect_ratio_range = aspect_ratio_range
def grayscale(self, rgb):
return rgb.dot([0.299, 0.587, 0.114])
def saturation(self, rgb):
gs = self.grayscale(rgb)
alpha = 2 * np.random.random() * self.saturation_var
alpha += 1 - self.saturation_var
rgb = rgb * alpha + (1 - alpha) * gs[:, :, None]
return np.clip(rgb, 0, 255)
def brightness(self, rgb):
alpha = 2 * np.random.random() * self.brightness_var
alpha += 1 - self.saturation_var
rgb = rgb * alpha
return np.clip(rgb, 0, 255)
def contrast(self, rgb):
gs = self.grayscale(rgb).mean() * np.ones_like(rgb)
alpha = 2 * np.random.random() * self.contrast_var
alpha += 1 - self.contrast_var
rgb = rgb * alpha + (1 - alpha) * gs
return np.clip(rgb, 0, 255)
def lighting(self, img):
cov = np.cov(img.reshape(-1, 3) / 255.0, rowvar=False)
eigval, eigvec = np.linalg.eigh(cov)
noise = np.random.randn(3) * self.lighting_std
noise = eigvec.dot(eigval * noise) * 255
img += noise
return np.clip(img, 0, 255)
def horizontal_flip(self, img, y):
if np.random.random() < self.hflip_prob:
img = img[:, ::-1]
y[:, [0, 2]] = 1 - y[:, [2, 0]]
return img, y
def vertical_flip(self, img, y):
if np.random.random() < self.vflip_prob:
img = img[::-1]
y[:, [1, 3]] = 1 - y[:, [3, 1]]
return img, y
def random_sized_crop(self, img, targets):
img_w = img.shape[1]
img_h = img.shape[0]
img_area = img_w * img_h
random_scale = np.random.random()
random_scale *= (self.crop_area_range[1] -
self.crop_area_range[0])
random_scale += self.crop_area_range[0]
target_area = random_scale * img_area
random_ratio = np.random.random()
random_ratio *= (self.aspect_ratio_range[1] -
self.aspect_ratio_range[0])
random_ratio += self.aspect_ratio_range[0]
w = np.round(np.sqrt(target_area * random_ratio))
h = np.round(np.sqrt(target_area / random_ratio))
if np.random.random() < 0.5:
w, h = h, w
w = min(w, img_w)
w_rel = w / img_w
w = int(w)
h = min(h, img_h)
h_rel = h / img_h
h = int(h)
x = np.random.random() * (img_w - w)
x_rel = x / img_w
x = int(x)
y = np.random.random() * (img_h - h)
y_rel = y / img_h
y = int(y)
img = img[y:y+h, x:x+w]
new_targets = []
for box in targets:
cx = 0.5 * (box[0] + box[2])
cy = 0.5 * (box[1] + box[3])
if (x_rel < cx < x_rel + w_rel and
y_rel < cy < y_rel + h_rel):
xmin = (box[0] - x_rel) / w_rel
ymin = (box[1] - y_rel) / h_rel
xmax = (box[2] - x_rel) / w_rel
ymax = (box[3] - y_rel) / h_rel
xmin = max(0, xmin)
ymin = max(0, ymin)
xmax = min(1, xmax)
ymax = min(1, ymax)
box[:4] = [xmin, ymin, xmax, ymax]
new_targets.append(box)
new_targets = np.asarray(new_targets).reshape(-1, targets.shape[1])
return img, new_targets
def generate(self, train=True):
while True:
if train:
shuffle(self.train_keys)
keys = self.train_keys
else:
shuffle(self.val_keys)
keys = self.val_keys
inputs = []
targets = []
for key in keys:
img_path = self.path_prefix + key
img = imread(img_path).astype('float32')
y = self.gt[key].copy()
if train and self.do_crop:
img, y = self.random_sized_crop(img, y)
img = imresize(img, self.image_size).astype('float32')
if train:
shuffle(self.color_jitter)
for jitter in self.color_jitter:
img = jitter(img)
if self.lighting_std:
img = self.lighting(img)
if self.hflip_prob > 0:
img, y = self.horizontal_flip(img, y)
if self.vflip_prob > 0:
img, y = self.vertical_flip(img, y)
y = self.bbox_util.assign_boxes(y)
inputs.append(img)
targets.append(y)
if len(targets) == self.batch_size:
tmp_inp = np.array(inputs)
tmp_targets = np.array(targets)
inputs = []
targets = []
yield preprocess_input(tmp_inp), tmp_targets
path_prefix = 'C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/JPEGImages/'
gen = Generator(gt, bbox_util, 16, 'C:/Users/in0131/Downloads/data(1)/VOCtrainval_06-Nov-2007/VOCdevkit/VOC2007/JPEGImages/',
train_keys, val_keys,
(input_shape[0], input_shape[1]), do_crop=False)
model = SSD300(input_shape, num_classes=NUM_CLASSES)
model.load_weights('weights_SSD300.hdf5', by_name=True)
freeze = ['input_1', 'conv1_1', 'conv1_2', 'pool1',
'conv2_1', 'conv2_2', 'pool2',
'conv3_1', 'conv3_2', 'conv3_3', 'pool3']#,
# 'conv4_1', 'conv4_2', 'conv4_3', 'pool4']
for L in model.layers:
if L.name in freeze:
L.trainable = False
def schedule(epoch, decay=0.9):
return base_lr * decay**(epoch)
callbacks = [keras.callbacks.ModelCheckpoint('./checkpoints/weights.{epoch:02d}-{val_loss:.2f}.hdf5',
verbose=1,
save_weights_only=True),
keras.callbacks.LearningRateScheduler(schedule)]
base_lr = 3e-4
optim = keras.optimizers.Adam(lr=base_lr)
# optim = keras.optimizers.RMSprop(lr=base_lr)
# optim = keras.optimizers.SGD(lr=base_lr, momentum=0.9, decay=decay, nesterov=True)
model.compile(optimizer=optim,
loss=MultiboxLoss(NUM_CLASSES, neg_pos_ratio=2.0).compute_loss)
nb_epoch = 30
history = model.fit_generator(gen.generate(True), gen.train_batches,
nb_epoch, verbose=1,
callbacks=callbacks,
validation_data=gen.generate(False),
nb_val_samples=gen.val_batches,
nb_worker=1)
@liu6381810 No problem. Actually, because of the size of my images I had the problem of small detection, but in my case it was about (6x6) in the (300x300).
The main problem you will have is that the smallest detection ratio is of 0.1 i.e 30 pixels, in order to correct that you may want to add an other detection layer after one layer inside the classifier but you will have to generate again the boxes.
Also keep in mind that the input image is 300x300 thus for too small objects you will lose most of the features, so you may want to add layers to be able to input bigger images (if possible, maybe your set is already sized 300x300).
And have you tried to lower the confidence threshold when using your classifier ? I had to go at 0.3 to get results with my network.
@ManjeeraJagiri I don't see any modification in the original model so model.save() and load_model() should work both fine (with those method you will have both the model and weights saved and loaded). Also the new weights will be saved after each epoch in the checkpoints directory :
callbacks = [keras.callbacks.ModelCheckpoint('./checkpoints/weights.{epoch:02d}-{val_loss:.2f}.hdf5',
verbose=1,
save_weights_only=True),
keras.callbacks.LearningRateScheduler(schedule)]
EDIT : You can get the model.save() and load_model() to work but if you want the easy way just use the weights.
@D3lt4lph4 If your object just occupy 6 * 6 pixel ,so how do you modify your network? my object is about 15 * 20 , even that I can't train my network correctly. I have scale the priors so the priors' height and width is just 1/2 of the original priors' and also modify the priorbox = PriorBox(img_size, 30.0, aspect_ratios=[2], variances=[0.1, 0.1, 0.2, 0.2], name='conv4_3_norm_mbox_priorbox') 30.0 to 15.0 and all priorboxs I have change to half of the original and so is variance.
EDIT: I just have 1-3 priors for my ground truth, is there some problem with this?
@liu6381810 , so basically, these lines of code below will create a .hdf5 file which stores the new weights in a new folder 'checkpoints'. Did I get that part right?
callbacks = [keras.callbacks.ModelCheckpoint('./checkpoints/weights.{epoch:02d}-{val_loss:.2f}.hdf5',
verbose=1,
save_weights_only=True),
keras.callbacks.LearningRateScheduler(schedule)]
So in SSD.ipynb, I replace the following part of the code model.load_weights('weights_SSD300.hdf5', by_name=True), with my new weights file i.e model.load_weights('weights.hdf5', by_name=True)
and leave this part model = SSD300(input_shape, num_classes=NUM_CLASSES) as it is?
Also,I dont understand why by_name is set as True, when according to keras documentation, you set it as True only when the you need to load weights into a different architecture (with some layers in common), for instance for fine-tuning or transfer-learning. Even though we are doing fine tuning, we haven't changed the architecture. So it should be set as False right?
@ManjeeraJagiri
Yes
1.you should make sure that you have ./checkpoints directory or it doesn't save weights sometime
2.model = SSD300(input_shape, num_classes=NUM_CLASSES) model.load_weights('your_weights.hdf5', by_name=True)
can lead you to get the model with your weight
3.in this code,both True or False are ok
@liu6381810 Thanks a lot! I will create a checkpoints directory. And do we have to change the file 'prior_boxes_ssd300.pkl' for custom datasets? Any idea what it actually contains and does?
@liu6381810 I did two things, first I added layers before the VGG16 to be able to input bigger images 1210x1210 (the idea was to be able to extract features more easily on small objects) . And second I added an other Priorbox layer on the conv3_2 layer (not sure about the layer name and I don't have access to the code right now) with smaller boxes.
The thing is, you have to modify 'prior_boxes_ssd300.pkl' file if you modify the PriorBox layers, because it is used to do the matching ground truth box/prior and you would end up with an incorrect matching.
I am not sure what you mean exactly with "1-3 priors for my ground truth", but if it is that you have only 1-3 box(es) per image, I cannot really tell. I had few objects per image (1-2 per class), but they were with very characteristic features so it was not much of a problem. I don't have any metric to have an idea of the number of boxes you would need to get good results. The only thing you can do is train your network and look for clues of why it is working/not working (whether it is the size of your dataset or the network itself that is wrong)
@ManjeeraJagiri 'prior_boxes_ssd300.pkl' file contains all of the prior boxes (maybe variances too) and is used by the Generator through the BBoxUtility to generate the y_true later used for the update of the weights. And so as long as you do not change the network (other than changing the number of classes), you don't need to change it.
@liu6381810 @D3lt4lph4 , The training got completed and saved the new weights successfully! Thanks a lot for your help. It works on videotest_example.py but when I run it on images, I get the following image
I am not very concerned about the accuracy of the boxes as I ran it only for 1 epoch instead on 30(I am running the code on a CPU so I reduced it. I just wanted to check if the code is running) Also, when I ran SSD.ipynb with the given weights i.e weights_SSD300.hdf5, I got the same following error:
@ManjeeraJagiri
Can you please share how was the results improved after you added this layer? Also I tried to add a layer as you did but end up with exceptions about dimension Can you please share your updated network structure?
Thanks
@Walid-Ahmed, Sorry but I have not added any layer. I just trained the same network again on PASCAL VOC data.
hi @D3lt4lph4,
You said:
The thing is, you have to modify 'prior_boxes_ssd300.pkl' file if you modify the PriorBox layers, because it is used to do the matching ground truth box/prior and you would end up with an incorrect matching.
The question is how you generate prior_boxes_ssd300.pkl file? Thank you.
