evopose2d icon indicating copy to clipboard operation
evopose2d copied to clipboard

Published 20 hours ago • verified publisher icon wmcnally

GPU training with tfrecords causes memory leak

Open naruto-raj opened this issue 3 years ago • 2 comments

I have tried to set batchsize 1 but still can see the memory usage slowly increasing until the process gets killed without any error or warning.

naruto-raj avatar Aug 29 '21 09:08 naruto-raj

This is a known issue and I do not know the cause. I did not experience it when training on TPU.

wmcnally avatar Aug 30 '21 11:08 wmcnally

Could you point to anything that would help me analyse this issue and find fixes.

naruto-raj avatar Sep 01 '21 13:09 naruto-raj