pedestrian-detection-ssdlite

pedestrian-detection-ssdlite copied to clipboard

total_loss = tf.check_numerics(total_loss, 'LossTensor is inf or nan.')

1

hello , I meet the below problem: total_loss = tf.check_numerics(total_loss, 'LossTensor is inf or nan.') the loss is nan. could you help me?

10183308

Out of memory when training the mobilenetv2-ssdlite?

1

cuda out of memory: the log is like this 2019-05-09 15:24:17.105701: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 6.25G (6712326144 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY 2019-05-09 15:24:17.534339: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 5.63G...

SyGoing