Anyone experience memory leak while training

[realwecan]

It seems to me that there may have been a memory leak during training, ultimately filling up all the available system RAM (not GPU memory) and causes the system to hang.

[HuangBo-Terraloupe]

Hello, check if there multiple Session are open. one more question: do you finish the evaluation code?

[realwecan]

@HuangBo-Terraloupe Check this one out: https://github.com/souryuu/FastMaskRCNN/tree/mask_visualization

[souryuu]

That is not a proper evaluation code. I only used it to visualize the mask network and groundtruth and while training. It is not for evaluating the final mask from test.

[souryuu]

For those who still have the problem about memory leak, I found that using "the tcmalloc allocator" solved the issue. https://stackoverflow.com/documentation/tensorflow/3883/how-to-debug-a-memory-leak-in-tensorflow#t=201612280142239281993

[realwecan]

Thanks @souryuu , could you please be able to share with us about your progress on the experiments? I see that you have obtained better performance with your codes, any chance we are getting closer to the results in the Mask RCNN paper? Thanks!