an illegal memory access was encountered

[FishBigOcean]

out of memory invalid argument an illegal memory access was encountered an illegal memory access was encountered 2019-08-12 01:46:36.265776: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:639] failed to record completion event; therefore, failed to create inter-stream dependency 2019-08-12 01:46:36.265776: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:639] failed to record completion event; therefore, failed to create inter-stream dependency 2019-08-12 01:46:36.265818: I tensorflow/stream_executor/stream.cc:4138] stream 0x5580c77c5e60 did not memcpy host-to-device; source: 0x7fbeda64a010 2019-08-12 01:46:36.265800: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:639] failed to record completion event; therefore, failed to create inter-stream dependency 2019-08-12 01:46:36.265814: I tensorflow/stream_executor/stream.cc:4138] stream 0x5580c77c5e60 did not memcpy host-to-device; source: 0x1020dc1a600 2019-08-12 01:46:36.265851: E tensorflow/stream_executor/stream.cc:289] Error recording event in stream: error recording CUDA event on stream 0x5580c77c5f30: CUDA_ERROR_ILLEGAL_ADDRESS; not marking stream as bad, as the Event object may be at fault. Monitor for further errors. 2019-08-12 01:46:36.265855: I tensorflow/stream_executor/stream.cc:4138] stream 0x5580c77c5e60 did not memcpy host-to-device; source: 0x1020dc26200 2019-08-12 01:46:36.265872: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS 2019-08-12 01:46:36.265886: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1 已放弃 (核心已转储)

I have encountered this problem, my environment is installed according to the readme, is there anyone who has a solution?

[ashnair1]

@FishBigOcean Were you able to solve this?

[Pluto1314]

hello！When i use multi gpu, I also have the same problem，were you able to solve it？