faster_rcnn icon indicating copy to clipboard operation
faster_rcnn copied to clipboard

Problem in training with GeForce GTX1070 and CUDA 8.0

Open oneQuery opened this issue 8 years ago • 2 comments

Is there anybody who succeeded running 'script_faster_rcnn_VOC...' with GeForce GTX1070 and CUDA 8.0?

When I use GeForce GTX 960M and CUDA 6.5, it doesn't have any problem. But there is a problem as I use the GPU, GeForce GTX1070, and CUDA 8.0 with like this error:

Error using caffe_
glog check error, please check log and clear mex

Error in caffe.Solver/step (line 56)
caffe_('solver_step', self.hSolver_self, iters);

Error in fast_rcnn_train>check_gpu_memory (line 216)
caffe_solver.step(1);

Error in fast_rcnn_train (line 89)
check_gpu_memory(conf, caffe_solver, num_classes, opts.do_val);

Error in Faster_RCNN_Train.do_fast_rcnn_train (line 7)
model_stage.output_model_file = fast_rcnn_train(conf, dataset.imdb_train, dataset.roidb_train, ...

Error in script_faster_rcnn_VOC2007_ZF (line 53)
model.stage1_fast_rcnn = Faster_RCNN_Train.do_fast_rcnn_train(conf_fast_rcnn, dataset,
model.stage1_fast_rcnn, opts.do_val);

And this is the log:

.
.
.
I1211 21:28:12.254725    96 net.cpp:743] Ignoring source layer fc8
I1211 21:28:12.254725    96 net.cpp:743] Ignoring source layer prob
F1211 21:28:42.700903    96 math_functions.cu:422] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE
F1211 21:28:42.700903    96 math_functions.cu:422] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE_

I think it has a problem with GPU, GPU driver and CUDA version. So I'm wondering is there anybody who succeeded running 'script_faster_rcnn_VOC...' with GeForce GTX1070 and CUDA 8.0 (I'm using MATLAB 2016b and Window 8.1)

oneQuery avatar Dec 11 '16 13:12 oneQuery

@assess09 Have you solved it now?

xzabg avatar Dec 27 '16 11:12 xzabg

GTX 1080 can't use CUDNN 5, recompile with that disabled. Use CUDA 7.5. Enable PTX.

rtgoring avatar Jan 04 '17 00:01 rtgoring