Check failed: error == cudaSuccess (2 vs. 0) out of memory

[Misaka299]

PS H:\WorkSpace\openpose> .\bin\OpenPoseDemo.exe --image_dir .\in\ --face --hand --write_images .\out
Starting OpenPose demo... Configuring OpenPose... Starting thread(s)... Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0. F0922 01:52:35.655333 12712 math_functions.cu:79] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** PS H:\WorkSpace\openpose> nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Sun_Aug_15_21:18:57_Pacific_Daylight_Time_2021 Cuda compilation tools, release 11.4, V11.4.120 Build cuda_11.4.r11.4/compiler.30300941_0 PS H:\WorkSpace\openpose> nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Sun_Aug_15_21:18:57_Pacific_Daylight_Time_2021 Cuda compilation tools, release 11.4, V11.4.120 Build cuda_11.4.r11.4/compiler.30300941_0 PS H:\WorkSpace\openpose>

How to solve this problem. The log level is adjusted to 1 and there is no more related error information. There is no problem running the file in the example. The picture is a jpg format transferred from the photoshop. Memory and video memory are completely sufficient.The occupancy has not changed too high while running.

CPU: i5 9600KF GPU: RTX 2060 Super

OpenPose version 1.7.0 (GPU release), from:

• https://github.com/CMU-Perceptual-Computing-Lab/openpose/releases/tag/v1.7.0

[italosalgado14]

I have the same problem with cuda 11 to run the code. This happens because of the incompatibility of cudnn 8, which in turn we are forced to use if we install cuda 11. have you been able to solve it? (what version of cudnn you use?)

[hadi1376tm]

same here

[nianfengnuan]

如何调整日志级别呢

[PanXiebit]

@italosalgado14

I have the same problem with cuda 11 to run the code. This happens because of the incompatibility of cudnn 8, which in turn we are forced to use if we install cuda 11. have you been able to solve it? (what version of cudnn you use?)

I meet the same error, I use 3090, and my cuda is 11.0, cudnn is 8.2.

Have you solved the problem?

[Misaka299]

I have tried different versions of cudaa. I didn't solve it, but later found out that I didn't use this function. This issue has been put on hold by me. Even so, there are still people who have encountered the same problem but have not solved it. Should I close this issues?

[italosalgado14]

I use them for easymocap. Try to deactivate the flag of cudnn in the compilation/instalation. CudNN of openpose dont work well with cuda 11. If you activate all the compilation errors, you can see the error is for cudnn dont resolve references. I use cuda 11.4 in a nvidia 3070 and works with cudnn deactivate in flags of openpose (deactivate in the menu of cmake installation), but the use of memory is twice in my GPU (4Gbs ).

[yakovkeselman]

Below is code that works in Google Colab. You can check their combination of various parameters (cudnn, nvidia-toolkit, cuda). On a Google cloud GPU-enabled machine what worked was libnvidia-470, libcudnn7, cuda-11.3. In particular, I only needed to plug libcudnn7 for libcudnn8.

!apt list --installed libnvidia*
!apt list --installed "libcudnn*"
!apt list --installed cuda-cudart*

import os
from os.path import exists, join, basename, splitext

git_repo_url = 'https://github.com/CMU-Perceptual-Computing-Lab/openpose.git'
project_name = splitext(basename(git_repo_url))[0]

!rm -rf openpose
# see: https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/949
print("install new CMake becaue of CUDA10")
cmake_version = 'cmake-3.20.2-linux-x86_64.tar.gz'
if not exists(cmake_version):
    !wget -q 'https://cmake.org/files/v3.20/{cmake_version}'
!tar xfz {cmake_version} --strip-components=1 -C /usr/local

print("clone openpose")
!git clone -q --depth 1 $git_repo_url
print("install system dependencies")
!apt-get -qq install -y libatlas-base-dev libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libgflags-dev libgoogle-glog-dev liblmdb-dev opencl-headers ocl-icd-opencl-dev libviennacl-dev
print("build openpose")
!cd /content/openpose && rm -rf build || true && mkdir build && cd build && cmake -DBUILD_PYTHON=1 .. && make -j`nproc`

[ofir1080]

Try to disable CuDNN changing USE_CUDNN=OFF and remake.

[lauraplaza2001]

Try to disable CuDNN changing USE_CUDNN=OFF and remake.

but how can you change USE_CUDNN ? where can you execute it?

[hadi1376tm]

Try to disable CuDNN changing USE_CUDNN=OFF and remake.

but how can you change USE_CUDNN ? where can you execute it?

By using CMake.

[tranzmatt]

Can confirm for me that disabling CUDNN fixed me issue. It'd run fine with the base openpose, but blow up if I tried to add --face or --hand. Disabling allows it to run. Alas, that seems to come at a real performance cost.