GPU is mostly idle when IMS_PER_BATCH is set to 32 on a large dataset (20w images)

[ronnie-tian]

Hi all,

I'm training fast rcnn with edge box proposal on a dataset which has around 20w images. Since most of the images only has one labelled ground truth bounding box, when I set IMS_PER_BATCH = 4, and BATCH_SIZE = 128, the training is ok (~0.2s/iter), GPU is always in use (>85% usage). But when I set IMS_PER_BATCH = 32 and BATCH_SIZE = 128, the training is very slow (~3s/iter) and most of the time GPU usage is 0%. I suspect the program spend a lot time preparing the mini_batch in ROI layer since the image number is very large. Is this normal? Any suggestions to this problem? Thank you.

[ranju2015]

@shawn-tian Can I have your model file with Edge box proposal? Why did you choose Edge box over RPN? Please explain if possible?