Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM

[nyanmn]

Installation is fine. During the installation, I had one issue with cudnn5.1 and I followed the suggestion here and now installation is fine.

Now I test the demo code as ./tools/demo.py

Then I have error as

I1117 09:48:41.011925 12503 net.cpp:51] Initializing net from parameters: 
name: "VGG_ILSVRC_16_layers"
state {
  phase: TEST
  level: 0
}
.
.
.
layer {
  name: "cls_prob"
  type: "Softmax"
  bottom: "cls_score"
  top: "cls_prob"
}
I1117 09:48:41.012234 12503 layer_factory.hpp:77] Creating layer input
I1117 09:48:41.012251 12503 net.cpp:84] Creating Layer input
I1117 09:48:41.012259 12503 net.cpp:380] input -> data
I1117 09:48:41.012271 12503 net.cpp:380] input -> im_info
I1117 09:48:41.328574 12503 net.cpp:122] Setting up input
I1117 09:48:41.328608 12503 net.cpp:129] Top shape: 1 3 224 224 (150528)
I1117 09:48:41.328614 12503 net.cpp:129] Top shape: 1 3 (3)
I1117 09:48:41.328618 12503 net.cpp:137] Memory required for data: 602124
I1117 09:48:41.328624 12503 layer_factory.hpp:77] Creating layer conv1_1
I1117 09:48:41.328655 12503 net.cpp:84] Creating Layer conv1_1
I1117 09:48:41.328660 12503 net.cpp:406] conv1_1 <- data
I1117 09:48:41.328670 12503 net.cpp:380] conv1_1 -> conv1_1
F1117 09:48:41.676553 12503 cudnn.hpp:128] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0)  CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
Aborted (core dumped)

What is wrong with my installation for this faster rcnn?

I have cuda8.0 and libcudnn5_5.1.10-1+cuda8.0 is installed on Ubuntu16.04. I have Qurdo K4200 graphic card.

[cechung]

I encountered the same problem, and I use cuda8.0 & cudnn5.1.10 & Ubuntu14.04 & 1080Ti. Is there anybody can give us solutions?

I have tried this and it doesn't work. https://github.com/rbgirshick/py-faster-rcnn/issues/237

[Huangswust182]

@EricccChung I also have the same problem, and if you've solved this problem, can you tell me how to solve it? thank you very much.

[nyanmn]

I have this solution, please have a look.

https://stackoverflow.com/questions/47342267/check-failed-status-cudnn-status-success-3-vs-0-cudnn-status-bad-param-fo

On Mon, Nov 20, 2017 at 1:12 PM, EricccChung [email protected] wrote:

I encountered the same problem, and I use cuda8.0 & cudnn5.1.10 & Ubuntu14.04 & 1080Ti. Is there anybody can give us solutions?

I have tried this and it doesn't work. #237 https://github.com/rbgirshick/py-faster-rcnn/issues/237

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rbgirshick/py-faster-rcnn/issues/733#issuecomment-345593549, or mute the thread https://github.com/notifications/unsubscribe-auth/AOJcY6XJ-G8ICwSYPK7AR_1VdwsErs3Vks5s4Qo1gaJpZM4QhdrO .

[nyanmn]

I have this solution, please have a look.

https://stackoverflow.com/questions/47342267/check-failed-status-cudnn-status-success-3-vs-0-cudnn-status-bad-param-fo

On Mon, Nov 20, 2017 at 7:48 PM, Huangswust182 [email protected] wrote:

@EricccChung https://github.com/ericccchung I also have the same problem, and if you've solved this problem, can you tell me how to solve it? thank you very much.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rbgirshick/py-faster-rcnn/issues/733#issuecomment-345673206, or mute the thread https://github.com/notifications/unsubscribe-auth/AOJcYzPAYzkLatoQKeD__Xtrwx60NOUwks5s4WcfgaJpZM4QhdrO .

[Huangswust182]

@nyanmn Thank you for your answer,but caffe-fast-rcnn branch doesn't support cundnnV6?

[nyanmn]

Not sure, but I can use with v6.

Sent from my iPhone

On 20 Nov 2017, at 8:26 PM, Huangswust182 [email protected] wrote:

@nyanmn Thank you for your answer,but caffe-fast-rcnn branch doesn't support cundnnV6?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nyanmn]

You will need to replace with main trunk of caffe, I did that.

Sent from my iPhone

On 20 Nov 2017, at 8:26 PM, Huangswust182 [email protected] wrote:

@nyanmn Thank you for your answer,but caffe-fast-rcnn branch doesn't support cundnnV6?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[cechung]

@nyanmn You mean that replace whole folder(caffe-fast-rcnn) at $FRCN_ROOT with folder downloaded from caffe github?

[nyanmn]

I did this first

cd caffe-fast-rcnn git remote add caffe https://github.com/BVLC/caffe.git git fetch caffe git merge -X theirs caffe/master

Remove self_.attr("phase") = static_cast(this->phase_); from include/caffe/layers/python_layer.hpp after merging.

Then upgrade to cudnnv6.0 for CUDA8.0, it worked for me.

thanks

On Tue, Nov 21, 2017 at 12:56 PM, EricccChung [email protected] wrote:

@nyanmn https://github.com/nyanmn You mean that replace whole folder(caffe-fast-rcnn) at $FRCN_ROOT with folder downloaded from caffe github?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rbgirshick/py-faster-rcnn/issues/733#issuecomment-345916875, or mute the thread https://github.com/notifications/unsubscribe-auth/AOJcY4RpoMPFIu4SCBAKtcrxcF6ee6d1ks5s4lfsgaJpZM4QhdrO .

[cechung]

OK I will try it. I will reply if it does work. Thanks a lot!

[nyanmn]

Please refer to my link for detail https://stackoverflow.com/questions/47342267/check-failed-status-cudnn-status-success-3-vs-0-cudnn-status-bad-param-fo pls vote if it works.

On Tue, Nov 21, 2017 at 1:10 PM, EricccChung [email protected] wrote:

OK I will try it. I will reply if it does work. Thanks a lot!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rbgirshick/py-faster-rcnn/issues/733#issuecomment-345918741, or mute the thread https://github.com/notifications/unsubscribe-auth/AOJcY8fHYu0pwCwnedBMVydPtOe_SXnmks5s4ls1gaJpZM4QhdrO .

[Huangswust182]

@nyanmn Hello, I can't use cudnnv6+cuda8.0 to compile caffe-fast-rcnn, and I've made a lot of mistakes, for example Make: * * [.build_release/src/caffe/syncedmem.o] Error 1, etc. Is there anything else that needs to be changed?

[Jingwei-Liao]

i have just solved my problem by adding engine: CAFFE in the convolution_param you can try this

[napolun279]

@woshiljw Hi，I want to know which document I should put in "engine: CAFFE" ？ thanks!!!

[Jingwei-Liao]

@napolun279 in the discription file of net you should add "engine: CAFFE" in "convolution_param" which belong to Convolution layer (my english is not so good hope you can understand)

[napolun279]

@woshiljw
thank you， I will have a try, and then I will give you a reply！

[Jingwei-Liao]

@napolun279 you should add this in every Convolution layer

[napolun279]

@woshiljw
ok, thanks very much for your enthusiastic answer！！！

[Jingwei-Liao]

@napolun279 you are welcome,hope you can also help more people～

[trikim]

@nyanmn Hi, I have tried your solution, but it doesn't work. The error was changed from F1212 22:23:30.527931 21847 cudnn_relu_layer.cu:24] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM to Segmentation fault (core dumped).

[nyanmn]

That could be some other issue. That is relu layer issue.

Sent from my iPhone

On 12 Dec 2017, at 10:49 PM, trikim [email protected] wrote:

@nyanmn Hi, I have tried your solution, but it doesn't work. The error was changed from F1212 22:23:30.527931 21847 cudnn_relu_layer.cu:24] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM to Segmentation fault (core dumped).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[trikim]

By reference from http://blog.csdn.net/u012841667/article/details/53436615, I changed all the cudnn* files, with newest caffe project from https://github.com/BVLC/caffe. the cudnn* files are in /py-faster-rcnn/caffe-fast-rcnn/include/caffe/util/ , /py-faster-rcnn/caffe-fast-rcnn/include/caffe/layers/, /py-faster-rcnn/caffe-fast-rcnn/src/caffe/util/ and /py-faster-rcnn/caffe-fast-rcnn/src/caffe/layers/ my GPU is 940MX, memory is 2G. So the problem was changed to: Check failed: error == cudaSuccess (2 vs. 0) out of memory At last, I changed to use CPU mode. And it worked well by the command: ./tools/demo.py --cpu

[nyanmn]

I think you need to change your gpu to test faster rcnn. You can check gpu requirements in the website. I think need more than 2G. I don't remember exactly

Sent from my iPhone

On 13 Dec 2017, at 11:03 AM, trikim [email protected] wrote:

By reference from http://blog.csdn.net/u012841667/article/details/53436615, I changed all the cudnn* files, with newest caffe project from https://github.com/BVLC/caffe. the cudnn* files are in /py-faster-rcnn/caffe-fast-rcnn/include/caffe/util/ , /py-faster-rcnn/caffe-fast-rcnn/include/caffe/layers/, /py-faster-rcnn/caffe-fast-rcnn/src/caffe/util/ and /py-faster-rcnn/caffe-fast-rcnn/src/caffe/layers/ my GPU is 940MX, memory is 2G. So the problem was changed to: Check failed: error == cudaSuccess (2 vs. 0) out of memory At last, I changed to use CPU mode. And it worked well by the command: ./tools/demo.py --cpu

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tongpinmo]

@napolun279 @EricccChung @Huangswust182 @woshiljw @nyanmn my problem is when I run the demo.py Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR *** Check failure stack trace: *** Aborted i don't know how to solve it for several days!! I have changed all the cudnn* files, and the nvidia message is Sat Dec 23 15:54:13 2017
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 384.90 Driver Version: 384.90 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX 106... Off | 00000000:01:00.0 On | N/A | | 49% 31C P5 8W / 120W | 264MiB / 6071MiB | 2% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1149 G /usr/lib/xorg/Xorg 199MiB | | 0 1884 G cinnamon 62MiB | +-----------------------------------------------------------------------------+

[Brad-Robbins]

Was that initial attempt running as a test run? That sounds familiar to other issues I've come across.

[amlandas78]

To solve this problem you have to use CuDNN..But there is a issue related to CuDNN v5 or v5.1. Downgrade to v4 and rebuild caffe..I hope that ths issue should be solved..

Thank you.

[yjyjkim]

@woshiljw i've solved!!!! it finally works!!!! thank you!!

[shriyashchougule]

i have just solved my problem by adding engine: CAFFE in the convolution_param you can try this

It worked for me, wondering what does this parameter conveys ?

[rongduo]

@woshiljw It also worked for me. Thanks a lot. Happy new year!

[sljlp]

The codes written by others can run successfully on my compute but the codes by myself can't. When I run my scripts this error turns.so i think there's something wrong in my codes rather than in the cudnn or cuda. but i don't know where the problem is.