out of memory in GTX1080

[minbinL]

I trained this model on a machine that has GTX1080 and 16 GB memory.It always ends up with:

out of memory invalid argument an illegal memory access was encountered E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:662] failed to record completion event; therefore, failed to create inter-stream dependency I tensorflow/stream_executor/stream.cc:3788] stream 0x50b3390 did not memcpy host-to-device; source: 0x7f1c216f6e60 E tensorflow/stream_executor/stream.cc:272] Error recording event in stream: error recording CUDA event on stream 0x50989c0: CUDA_ERROR_ILLEGAL_ADDRESS; not marking stream as bad, as the Event object may be at fault. Monitor for further errors. E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:198] Unexpected Event status: 1 [1] 6900 abort (core dumped) python ./faster_rcnn/train_net.py --gpu 0 --weights --imdb voc_2007_trainval

and README.md said: Requirements: hardware For training the end-to-end version of Faster R-CNN with VGG16, 3G of GPU memory is sufficient (using CUDNN)

what's the problem?

[ghost]

I trained this model on a machine that has GTX1060 and 6 GB memory.It always runs normal.

[riadhayachi]

minimum memory to use is 4GB

[fangyan93]

@minbinL Hello, have you solved this problem? I am also using a GTX1080 to train this model on my dataset, this error occurs in the middle of training, and I trained it successfully once, but it keeps showing up now, any idea?

[baizhenmao95]

Oh, I meet the same problem with you. Did you solve it? I'll verrrrrrrrrrry appreciate for your help!@minbinL