ghost-free-shadow-removal icon indicating copy to clipboard operation
ghost-free-shadow-removal copied to clipboard

Training OOM & TF2?

Open glarfomatic opened this issue 2 years ago • 2 comments

Hi -

I'm using my own dataset at 640x480 to train, and I run out of memory almost immediately on my GPU, an RTX 2070 Super - I get a ton of memory allocation and OOM errors related to the GPU. Is there a way to mitigate this by changing batch sizes, etc?

Any update on moving this to TF2.x?

glarfomatic avatar Feb 11 '22 14:02 glarfomatic

closed by mistake

glarfomatic avatar Feb 11 '22 14:02 glarfomatic

Hi, you can try to remove some feature layers(currently 1000+) in the VGG19 backbone, which might not influence if you reduce some of them.

vinthony avatar Mar 17 '22 16:03 vinthony