LFFD-A-Light-and-Fast-Face-Detector-for-Edge-Devices icon indicating copy to clipboard operation
LFFD-A-Light-and-Fast-Face-Detector-for-Edge-Devices copied to clipboard

the speed of training

Open huanpig opened this issue 5 years ago • 4 comments

it's very slow while training the face detection,what can i do to speed up the training? the main reason for this is the pkl ,not rec for mxnet?

huanpig avatar Nov 05 '19 09:11 huanpig

@huanpig pkl format is much faster than rec of mxnet. pkl files are loaded into the memory, avoiding hard disk IO. The speed is determined by factors: 1) jpg decode speed (cv2 with turbo-jpeg); 2) the image size and the input size; 3) the average number of objects in input images (for loops). So the CPU performance matters.

YonghaoHe avatar Nov 23 '19 13:11 YonghaoHe

in article,you say:We train 1,500,000 iterations and reduce the learning rate by multiplying 0.1 at iteration 600,000, 1,000,000, 1,200,000 and 1,400,000. The training time is about 5 days with two NVIDIA GTX1080TI, in my thought,it's very slow, is it uner normal circumstances?

huanpig avatar Nov 26 '19 05:11 huanpig

@YonghaoHe

huanpig avatar Nov 26 '19 05:11 huanpig

@huanpig indeed, the training speed is not fast. The data preparation is slow. The training speed is about 50 images/s.

YonghaoHe avatar Nov 26 '19 05:11 YonghaoHe