ResourceExhaustedError: OOM

[Timos-K]

I always get an ResourceExhaustedError: OOM error whenever using this code. I'm unable to use any batch size greater than 256. Can you point out which parts are the most memory intensive?

ResourceExhaustedError: OOM when allocating tensor with shape[510,510,510,510] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
	 [[{{node loss_5/merged_layer_neg_loss/batch_all_triplet_loss/ToFloat_1}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

[Pandoro]

The loss computation itself should take up a marginal amount of memory. Even though a square matrix of distances is computed, it is by far smaller than the typical feature tensors. As such, the network itself will be the expensive part. Even reducing the last feature vector sizes should have marginal influence on the memory footprint of the network and as such, the only thing you can really do is either shrink the images, which has a quite detrimental effect on the performance (see our paper) or use a network with a smaller memory footprint. Our LuNet was able to train with significantly bigger batch sizes due to taking up by far less memory.

On Tue, Mar 31, 2020 at 3:10 PM Timos Korres [email protected] wrote:

I always get an ResourceExhaustedError: OOM error whenever using this code. I'm unable to use any batch size greater than 256. Can you point out which parts are the most memory intensive?

ResourceExhaustedError: OOM when allocating tensor with shape[510,510,510,510] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu [[{{node loss_5/merged_layer_neg_loss/batch_all_triplet_loss/ToFloat_1}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/VisualComputingInstitute/triplet-reid/issues/94, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOJDTM5KENTNSO4ZLDUQXTRKHTVVANCNFSM4LXRG4KQ .