openpose icon indicating copy to clipboard operation
openpose copied to clipboard

CUDA 12 Support

Open juanmarin96 opened this issue 2 years ago • 6 comments

Is there any work related with CUDA 12 support soon? Type of Issue Question

juanmarin96 avatar Apr 28 '23 00:04 juanmarin96

I'm installing in Ubuntu 22.04 with this:

Thu Jun 29 06:59:30 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro RTX 6000                On  | 00000000:0B:00.0 Off |                  Off |
| 34%   40C    P8               3W / 260W |      1MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

working well for general use, but crashing with face or hand parameters

./build/examples/openpose/openpose.bin --video /home/raul/tmp/pruebaOpenpose/videos/prueba/2016-01-02_0000_US_CNN_Erin_Burnett_Out_Front_2838.53-2842.73_ago.mp4 -display 0 --render_pose 0    --write_json /home/raul/tmp/pruebaOpenpose/
Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
OpenPose demo successfully finished. Total time: 7.893274 seconds.
./build/examples/openpose/openpose.bin --video /home/raul/tmp/pruebaOpenpose/videos/prueba/2016-01-02_0000_US_CNN_Erin_Burnett_Out_Front_2838.53-2842.73_ago.mp4 -display 0 --render_pose 0 --face   --write_json /home/raul/tmp/pruebaOpenpose/
Starting OpenPose demo...
Configuring OpenPose...
Starting thread(s)...
Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.
F0629 07:01:08.567824  4451 cudnn_relu_layer.cpp:13] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0)  CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
    @     0x7f795c1d2b03  google::LogMessage::Fail()
    @     0x7f795c1da9d1  google::LogMessage::SendToLog()
    @     0x7f795c1d27c2  google::LogMessage::Flush()
    @     0x7f795c1d478f  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f795be108a0  caffe::CuDNNReLULayer<>::LayerSetUp()
    @     0x7f795bee7699  caffe::Net<>::Init()
    @     0x7f795bee9610  caffe::Net<>::Net()
    @     0x7f795c8cac6c  op::NetCaffe::initializationOnThread()
    @     0x7f795c875597  op::FaceExtractorCaffe::netInitializationOnThread()
    @     0x7f795c876ef6  op::FaceExtractorNet::initializationOnThread()
    @     0x7f795c912335  op::Worker<>::initializationOnThreadNoException()
    @     0x7f795c9124a8  op::SubThread<>::initializationOnThread()
    @     0x7f795c913eb0  op::Thread<>::initializationOnThread()
    @     0x7f795c9163fc  op::Thread<>::threadFunction()
    @     0x7f795c51e2b3  (unknown)
    @     0x7f795c28cb43  (unknown)
    @     0x7f795c31ea00  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)

I have installed cuda 11.7 and cuda 12.2

dpkg -l | grep cuda
ii  cuda-11-7                              11.7.1-1                                amd64        CUDA 11.7 meta-package
ii  cuda-cccl-11-7                         11.7.91-1                               amd64        CUDA CCCL
ii  cuda-cccl-12-2                         12.2.53-1                               amd64        CUDA CCCL
ii  cuda-command-line-tools-11-7           11.7.1-1                                amd64        CUDA command-line tools
ii  cuda-command-line-tools-12-2           12.2.0-1                                amd64        CUDA command-line tools
ii  cuda-compiler-11-7                     11.7.1-1                                amd64        CUDA compiler
ii  cuda-compiler-12-2                     12.2.0-1                                amd64        CUDA compiler
ii  cuda-cudart-11-7                       11.7.99-1                               amd64        CUDA Runtime native Libraries
ii  cuda-cudart-12-2                       12.2.53-1                               amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-11-7                   11.7.99-1                               amd64        CUDA Runtime native dev links, headers
ii  cuda-cudart-dev-12-2                   12.2.53-1                               amd64        CUDA Runtime native dev links, headers
ii  cuda-cuobjdump-11-7                    11.7.91-1                               amd64        CUDA cuobjdump
ii  cuda-cuobjdump-12-2                    12.2.53-1                               amd64        CUDA cuobjdump
ii  cuda-cupti-11-7                        11.7.101-1                              amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-12-2                        12.2.60-1                               amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-dev-11-7                    11.7.101-1                              amd64        CUDA profiling tools interface.
ii  cuda-cupti-dev-12-2                    12.2.60-1                               amd64        CUDA profiling tools interface.
ii  cuda-cuxxfilt-11-7                     11.7.91-1                               amd64        CUDA cuxxfilt
ii  cuda-cuxxfilt-12-2                     12.2.53-1                               amd64        CUDA cuxxfilt
ii  cuda-demo-suite-11-7                   11.7.91-1                               amd64        Demo suite for CUDA
ii  cuda-documentation-11-7                11.7.91-1                               amd64        CUDA documentation
ii  cuda-documentation-12-2                12.2.53-1                               amd64        CUDA documentation
ii  cuda-driver-dev-11-7                   11.7.99-1                               amd64        CUDA Driver native dev stub library
ii  cuda-driver-dev-12-2                   12.2.53-1                               amd64        CUDA Driver native dev stub library
ii  cuda-drivers                           535.54.03-1                             amd64        CUDA Driver meta-package, branch-agnostic
ii  cuda-drivers-535                       535.54.03-1                             amd64        CUDA Driver meta-package, branch-specific
ii  cuda-gdb-11-7                          11.7.91-1                               amd64        CUDA-GDB
ii  cuda-gdb-12-2                          12.2.53-1                               amd64        CUDA-GDB
ii  cuda-keyring                           1.1-1                                   all          GPG keyring for the CUDA repository
ii  cuda-libraries-11-7                    11.7.1-1                                amd64        CUDA Libraries 11.7 meta-package
ii  cuda-libraries-12-2                    12.2.0-1                                amd64        CUDA Libraries 12.2 meta-package
ii  cuda-libraries-dev-11-7                11.7.1-1                                amd64        CUDA Libraries 11.7 development meta-package
ii  cuda-libraries-dev-12-2                12.2.0-1                                amd64        CUDA Libraries 12.2 development meta-package
ii  cuda-memcheck-11-7                     11.7.91-1                               amd64        CUDA-MEMCHECK
ii  cuda-nsight-11-7                       11.7.91-1                               amd64        CUDA nsight
ii  cuda-nsight-12-2                       12.2.53-1                               amd64        CUDA nsight
ii  cuda-nsight-compute-11-7               11.7.1-1                                amd64        NVIDIA Nsight Compute
ii  cuda-nsight-compute-12-2               12.2.0-1                                amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-11-7               11.7.1-1                                amd64        NVIDIA Nsight Systems
ii  cuda-nsight-systems-12-2               12.2.0-1                                amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-11-7                         11.7.99-1                               amd64        CUDA nvcc
ii  cuda-nvcc-12-2                         12.2.91-1                               amd64        CUDA nvcc
ii  cuda-nvdisasm-11-7                     11.7.91-1                               amd64        CUDA disassembler
ii  cuda-nvdisasm-12-2                     12.2.53-1                               amd64        CUDA disassembler
ii  cuda-nvml-dev-11-7                     11.7.91-1                               amd64        NVML native dev links, headers
ii  cuda-nvml-dev-12-2                     12.2.81-1                               amd64        NVML native dev links, headers
ii  cuda-nvprof-11-7                       11.7.101-1                              amd64        CUDA Profiler tools
ii  cuda-nvprof-12-2                       12.2.60-1                               amd64        CUDA Profiler tools
ii  cuda-nvprune-11-7                      11.7.91-1                               amd64        CUDA nvprune
ii  cuda-nvprune-12-2                      12.2.53-1                               amd64        CUDA nvprune
ii  cuda-nvrtc-11-7                        11.7.99-1                               amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-12-2                        12.2.91-1                               amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-11-7                    11.7.99-1                               amd64        NVRTC native dev links, headers
ii  cuda-nvrtc-dev-12-2                    12.2.91-1                               amd64        NVRTC native dev links, headers
ii  cuda-nvtx-11-7                         11.7.91-1                               amd64        NVIDIA Tools Extension
ii  cuda-nvtx-12-2                         12.2.53-1                               amd64        NVIDIA Tools Extension
ii  cuda-nvvp-11-7                         11.7.101-1                              amd64        CUDA Profiler tools
ii  cuda-nvvp-12-2                         12.2.60-1                               amd64        CUDA Profiler tools
ii  cuda-opencl-12-2                       12.2.53-1                               amd64        CUDA OpenCL native Libraries
ii  cuda-opencl-dev-12-2                   12.2.53-1                               amd64        CUDA OpenCL native dev links, headers
ii  cuda-profiler-api-12-2                 12.2.53-1                               amd64        CUDA Profiler API
ii  cuda-runtime-11-7                      11.7.1-1                                amd64        CUDA Runtime 11.7 meta-package
ii  cuda-sanitizer-11-7                    11.7.91-1                               amd64        CUDA Sanitizer
ii  cuda-sanitizer-12-2                    12.2.53-1                               amd64        CUDA Sanitizer
ii  cuda-toolkit                           12.2.0-1                                amd64        CUDA Toolkit meta-package
ii  cuda-toolkit-11-7                      11.7.1-1                                amd64        CUDA Toolkit 11.7 meta-package
ii  cuda-toolkit-11-7-config-common        11.7.99-1                               all          Common config package for CUDA Toolkit 11.7.
ii  cuda-toolkit-11-config-common          11.8.89-1                               all          Common config package for CUDA Toolkit 11.
ii  cuda-toolkit-12-2                      12.2.0-1                                amd64        CUDA Toolkit 12.2 meta-package
ii  cuda-toolkit-12-2-config-common        12.2.53-1                               all          Common config package for CUDA Toolkit 12.2.
ii  cuda-toolkit-12-config-common          12.2.53-1                               all          Common config package for CUDA Toolkit 12.
ii  cuda-toolkit-config-common             12.2.53-1                               all          Common config package for CUDA Toolkit.
ii  cuda-tools-11-7                        11.7.1-1                                amd64        CUDA Tools meta-package
ii  cuda-tools-12-2                        12.2.0-1                                amd64        CUDA Tools meta-package
ii  cuda-visual-tools-11-7                 11.7.1-1                                amd64        CUDA visual tools
ii  cuda-visual-tools-12-2                 12.2.0-1                                amd64        CUDA visual tools
ii  libcudnn8                              8.9.2.26-1+cuda12.1                     amd64        cuDNN runtime libraries
ii  libcudnn8-dev                          8.9.2.26-1+cuda12.1                     amd64        cuDNN development libraries and headers

Any idea to solve it?

RaulKite avatar Jun 29 '23 07:06 RaulKite

I'm running into the same problem. This is confirmed only to happen on cuda 12.x?

tranzmatt avatar Oct 29 '23 02:10 tranzmatt