face_recognition Issues Using CUDA with Face Recognition on Jetson Nano

System Information

face_recognition version: 1.3.0
Python version: 3.6.9 (JetPack Pre-installed Version)
Operating System: Ubuntu 18.04 (Jetpack 4.6)

Description

Hi all,

For the past week or so, I have been trying to get Face Recognition and Dlib to work with CUDA on the Jetson Nano 4GB. I have attempted multiple installation combinations of Dlib and the Face Recognition library, but none have succeeded in enabling me to use the "cnn" model with the face_locations() function (each failing with different kinds of errors).

Note: In each of the below cases, any problem was exhibited regardless of the image loaded into the face_locations() function. Where Dlib was built from source, the Dlib compilation process always reported that "DLIB WILL USE CUDA". Installations of Dlib that could be successfully imported into Python always returned True and 1 when "dlib.DLIB_USE_CUDA" and "print(dlib.cuda.get_num_devices())" were entered into Python respectively.

Installation Process

Originally, I attempted to install Dlib version 19.22.1 via "pip3 install dlib" (the latest version at the time prior to the release of Dlib version 19.23). I didn't think that I needed to implement the fix in Dlib for Jetson Nano outlined in the "Jetson Nano installation instructions" article in the README, as the problem mentioned in the article had supposedly since been fixed in a past cuDNN/JetPack SDK update. The library downloaded successfully, but attempting to import the library was met with "Segmentation fault (core dumped)". Research suggested that I try downgrading the library to version 19.21.1 and, on doing so (using pip), was able to import the library without issue and use it with face_recognition.

I then tried to use the "cnn" model with the face_locations() function in a simple program to detect the position of a face in an image. Each time I ran the program, the GPU showed 0% usage, and it would either freeze the computer before failing with "Killed" in the terminal (usually after a few minutes), or completely freeze up the entire system, forcing me to perform a hard restart of the computer. Using other face_recognition functions, such as face_encodings() and compare_faces() (for facial recognition), or face_locations() with the default HOG-based model worked without error.

With the release of Dlib 19.23, I uninstalled Dlib 19.21.1 and tried installing Dlib 19.23 using pip, but was also met with "Segmentation fault (core dumped)" on import into Python, as with Dlib 19.22.1. To rule out not building Dlib from source as being an issue, I uninstalled 19.23 and built from source the previously working Dlib 19.21 release (without applying the old cuDNN "forward_algo = forward_best_algo" fix), but the "cnn" model still failed to run.

I then attempted to build 19.23 using NumPy 1.19.4 (instead of version 1.19.5 that was previously installed on my system and used for the above few Dlib installations) and exporting the "OPENBLAS_CORETYPE=ARMV8" environmental variable into the Terminal (suggestions from the internet to fix the "Segmentation fault" on import issue), with no success.

My current installation now consists of a built from source version of Dlib 19.21 (using NumPy 1.19.4), following the instructions to comment out "forward_algo = forward_best_algo" before compiling the package. On running the "cnn" model with face_locations(), the GPU, for the first time, showed GPU usage (99%), before the program crashed 15-20 seconds later with the following error:

Click the arrow to view the full Python Terminal crash output.

Traceback (most recent call last):
  File "face-recog-test.py", line 36, in <module>
    face_locations = face_recognition.face_locations(image, number_of_times_to_upsample=0, model="cnn")
  File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 119, in face_locations
    return [_trim_css_to_bounds(_rect_to_css(face.rect), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, "cnn")]
  File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 103, in _raw_face_locations
    return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudnnFindConvolutionBackwardDataAlgorithm( context(), (const cudnnFilterDescriptor_t)filter_handle, descriptor(dest_desc), (const cudnnConvolutionDescriptor_t)conv_handle, descriptor(data), num_possilbe_algorithms, &num_algorithms, perf_results.data()) in file /home/dev/dlib-19.21/dlib/cuda/cudnn_dlibapi.cpp:856. code: 4, reason: A call to cuDNN failed
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated

If anyone can provide me with some advice on how to proceed, it would be greatly appreciated.

Thank you

Jan 30 '22 11:01 ACRONN

It's little bit late but if you still have problem, try this way:

wget http://dlib.net/files/dlib-19.23.tar.bz2
tar jxvf dlib-19.23.tar.bz2
cd dlib-19.23/
mkdir build
cd build/
cmake ..
cmake --build .
cd ../
sudo python3 setup.py install
sudo pip3 install face_recognition

With jtop command you can see if GPU is in use sudo pip3 install jetson-stats or better:

python3
import dlib
print(dlib.DLIB_USE_CUDA)

and check if output is True

Good luck!

Apr 12 '22 21:04 JakubSzozda

import dlib Illegal instruction (core dumped) jetson@jetson4g:~/face_recognition/dlib-19.23$

Jul 29 '22 10:07 jhkim3217

Same issue here. Have you found the fix?

Mar 18 '23 15:03 aljohn0422

It's little bit late but if you still have problem, try this way:
wget http://dlib.net/files/dlib-19.23.tar.bz2
tar jxvf dlib-19.23.tar.bz2
cd dlib-19.23/
mkdir build
cd build/
cmake ..
cmake --build .
cd ../
sudo python3 setup.py install
sudo pip3 install face_recognition
With jtop command you can see if GPU is in use sudo pip3 install jetson-stats or better:
python3
import dlib
print(dlib.DLIB_USE_CUDA)
and check if output is True

Good luck!

Is it necessary to perform a memory swap?

Sep 13 '23 15:09 VieiraJardel

It's little bit late but if you still have problem, try this way:
wget http://dlib.net/files/dlib-19.23.tar.bz2
tar jxvf dlib-19.23.tar.bz2
cd dlib-19.23/
mkdir build
cd build/
cmake ..
cmake --build .
cd ../
sudo python3 setup.py install
sudo pip3 install face_recognition
With jtop command you can see if GPU is in use sudo pip3 install jetson-stats or better:
python3
import dlib
print(dlib.DLIB_USE_CUDA)
and check if output is True

Good luck!

I followed the compilation steps, and when I type "print(dlib.DLIB_USE_CUDA)", the response is "TRUE." However, when I attempt to run the code, I receive a "Segmentation fault (core dumped)" error.

I followed Paul's steps, and everything worked, but the frames per second (fps) were low, and it wasn't using the GPU. The version of dlib was 19.17. When I tried the more recent versions like 19.24, 19.23, 19.21, I encountered the same "Segmentation fault (core dumped)" message.

Paul: https://www.youtube.com/watch?v=yOdVnQuhfeI&list=PLGs0VKk2DiYxP-ElZ7-QXIERFFPkOuP4_&index=38

Nvidia forum https://forums.developer.nvidia.com/t/simple-accelerated-face-recognition/142679/19

I'm using numpy 1.19.4

Sep 13 '23 15:09 VieiraJardel

face_recognition face_recognition copied to clipboard

Issues Using CUDA with Face Recognition on Jetson Nano

System Information

Description

Installation Process

face_recognition
face_recognition copied to clipboard