training process is slow

[E12dward]

i think class Generator runs on cpu ,maybe this is why the training process is so slow . is there any solution ? how many time is taken within one epoch on your computer ? thanks

[tanakataiki]

Is threre any ways to run generator on GPUs? I also think generator has to have maybe more workers to speed up generation

https://keras.io/ja/models/sequential/#fit_generator

similar issue

#16

[E12dward]

hi , when using the pretrained weights file (ssdmobilenetv2) shared in google drive , the train-loss value has been reduced while the val-loss value is almost constant . after testing it on VOC2007 , i found the mAP value is only 53. Have you encountered this problem ? what should been done about training strategy?

[tanakataiki]

Is it the same as my repo? I havent tested map but can you share or PR Map tester and weights? I might find something if possible.

[E12dward]

https://drive.google.com/file/d/1zts95H6A4-lrRD0HAvLH2oNHA8CgU6m1/view?usp=sharing

[E12dward]

https://github.com/Adamdad/Object-Detection-Metrics

[E12dward]

i would be glad to receive your answer , thx!

[E12dward]

the code is almost the same as your repo , maybe there is something diffierent in SSD_training.ipnb . Perhaps you can find something wrong by running the training code.

[tanakataiki]

Thanks I cant immidiately find something good but I will someday. I assume the difference would be weight multiplier for each layer or initialization details that keras would not support + implementation details which is very deep, by the way Pull request with details is always welcome !

[tanakataiki]

@E12dward Oh... Is your drive file broken ? i couldnt unzip rar file. and maybe ipython and weight is enough to share

[E12dward]

https://drive.google.com/drive/folders/1c4XUoHl1q9PPxCCG94m4A7sjI2H8XnJo?usp=sharing

[E12dward]

@tanakataiki i am sorry , the link above should be useful .

[lchunleo]

any way to speed up the training?