Python pose API GPU utilisation

[kelvinxuande]

Hi! Thank you so much for releasing this - have been looking for this for a long time.

I have diligently followed your instructions on a Jetson Xavier NX and managed to successfully use the python API (specifically .solutions) for pose estimation to query the keypoints in an image. I would like to ask - in such a scenario, is the GPU enabled and utilised? i.e. are there seperate bazel build procedure/ configurations to enable and utilise the GPU?

Thanks and best regards.

[efthimia145]

Hello @kelvinxuande , since you managed to download it, i would like to ask you what version of Jetpack was installed in the NX. Thank you!

[kelvinxuande]

Hi @efthimia145 I don't think it matters that much here - vaguely recall using either jp4.4.1 or jp4.5.1. I expect it to work either way. Follow the instructions released here by @jiuqiant and post here if you face any issues!

[efthimia145]

Hello @kelvinxuande! Thanks for your answer. I have followed the instructions, and I keep getting the following error:

ERROR: /home/nvidia/mediapipe/mediapipe/modules/objectron/calculators/BUILD:156:11: Compiling mediapipe/modules/objectron/calculators/epnp.cc failed: (Exit 4): gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 48 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
gcc: internal compiler error: Segmentation fault signal terminated program cc1plus

As I understand, gcc: internal compiler error: Segmentation fault signal terminated program cc1plus is what causing the problem, but I do not understand what causes it.

I am trying to install in on Jetson Xavier NX, JP version 4.6.1. I have also tried multiple gcc versions (7.+), but did not help. Let me know if any other system information would be useful.

(FYI I had the same error when installing opencv, and I solved it by using only 1 core for the installation ( -j1), but I am not sure If something like that would help in this case.)

[kelvinxuande]

In luck! @efthimia145. Managed to dig out the installation steps I used. Can confirm it to work for jp4.4.1. Maybe do a cross-check to see if you've omitted any steps here that might help? Do get back on this - keen to see how the port over to jp4.6 might look like.

Steps to build MediaPipe Python on aarch64

Tested on Jetson Xavier NX, Jetpack4.4.1. Source

1. Build and setup Bazel from source, clone the mediaPipe repository.

   ### For Bazel 3.4.1
   $ cd Downloads/
   $ mkdir bazel-3.4.1
   $ cd bazel-3.4.1
   $ wget https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-dist.zip
   $ sudo apt-get install build-essential openjdk-8-jdk python zip unzip
   $ unzip bazel-3.4.1-dist.zip
   $ env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
   $ sudo cp output/bazel /usr/local/bin/
   ### clone the mediaPipe repository
   $ cd ~/Downloads/
   $ git clone https://github.com/google/mediapipe.git
   

2. Install build dependencies, system packages and prerequisites

   $ sudo apt-get install git cmake
   $ sudo apt-get install libatlas-base-dev gfortran
   $ sudo apt-get install libhdf5-serial-dev hdf5-tools
   $ sudo apt-get install python3-dev
   $ sudo apt-get install python3-testresources
   

3. Setup virtual environment

   ### Download and install pip
   $ wget https://bootstrap.pypa.io/get-pip.py
   $ sudo python3 get-pip.py
   $ rm get-pip.py
   ### Install virtualenv, virtualenvwrapper
   $ sudo apt install nano
   $ sudo pip install virtualenv virtualenvwrapper
   $ nano ~/.bashrc
   

   Scroll down to the bottom of the ~/.bashrc file and add the following lines:

   # virtualenv and virtualenvwrapper
   export WORKON_HOME=$HOME/.virtualenvs
   export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
   source /usr/local/bin/virtualenvwrapper.sh
   

   After adding the above lines, save and exit the editor.

   ### Reload contents of ~/.bashrc file
   $ source ~/.bashrc
   ### Make new virtualenv
   $ mkvirtualenv mediapipe -p python3
   

4. Install proto compiler.

   $ sudo apt install -y protobuf-compiler
   

5. Manually update protobuf compiler

   Download the latest protoc-3.x.x-linux-aarch_64.zip from GitHub.

   $ wget https://github.com/protocolbuffers/protobuf/releases/download/v3.14.0/protoc-3.14.0-linux-aarch_64.zip
   $ unzip protoc-3.14.0-linux-aarch_64.zip
   $ cd ~/Downloads/bin/
   $ sudo cp protoc /usr/local/bin/
   $ cd ~/Downloads/include/
   $ sudo cp -r google /usr/local/include/
   

   Extract and copy the contents in "bin" and "include/google" to system libraries.

   [TODO: directory tree]

6. Go to the MediaPipe directory.

   $ cd ~/Downloads/mediapipe
   

   Modify mediapipe/setup.py like the following:

   diff --git a/setup.py b/setup.py
   index 61848de..462d91d 100644
   --- a/setup.py
   +++ b/setup.py
   @@ -208,7 +208,7 @@ class GeneratePyProtos(setuptools.Command):
            sys.stderr.write('cannot find required file: %s\n' % source)
            sys.exit(-1)
   
   -      protoc_command = [self._protoc, '-I.', '--python_out=.', source]
   +      protoc_command = [self._protoc, '-I.', '-I/usr/local/include', '--python_out=.', source]
          if subprocess.call(protoc_command) != 0:
            sys.exit(-1)
   
   

7. Remove unnecessary OpenCV modules and linker flags.

   sed -i -e "/\"imgcodecs\"/d;/\"calib3d\"/d;/\"features2d\"/d;/\"highgui\"/d;/\"video\"/d;/\"videoio\"/d" third_party/BUILD
   sed -i -e "/-ljpeg/d;/-lpng/d;/-ltiff/d;/-lImath/d;/-lIlmImf/d;/-lHalf/d;/-lIex/d;/-lIlmThread/d;/-lrt/d;/-ldc1394/d;/-lavcodec/d;/-lavformat/d;/-lavutil/d;/-lswscale/d;/-lavresample/d" third_party/BUILD
   

8. Remove opencv-python from requirements.txt.

   diff --git a/requirements.txt b/requirements.txt
   index cee4e45..84617cb 100644
   --- a/requirements.txt
   +++ b/requirements.txt
   @@ -1,7 +1,6 @@
    absl-py
    dataclasses
   # -numpy == 1.19.3
   -opencv-python
   # +numpy
    protobuf>=3.11.4
    six
    wheel
   

9. Remove drawing_utils from __init__.py since opencv-python is not available on aarch64 Linux.

   --- a/mediapipe/python/solutions/__init__.py
   +++ b/mediapipe/python/solutions/__init__.py
   @@ -14,7 +14,8 @@
   
    """MediaPipe Solutions Python API."""
   
    -import mediapipe.python.solutions.drawing_utils
    +# Disable when opencv-python is not available.
    +# import mediapipe.python.solutions.drawing_utils
    import mediapipe.python.solutions.face_mesh
    import mediapipe.python.solutions.hands
    import mediapipe.python.solutions.holistic
   

10. Disable carotene_o4t in third_party/BUILD.

```
diff --git a/third_party/BUILD b/third_party/BUILD
index ef408e4..51e1104 100644
--- a/third_party/BUILD
+++ b/third_party/BUILD
@@ -110,6 +104,8 @@ cmake_external(
   "WITH_ITT": "OFF",
   "WITH_JASPER": "OFF",
   "WITH_WEBP": "OFF",
+   "ENABLE_NEON": "OFF",
+   "WITH_TENGINE": "OFF",
```

11. Build the package.

```bash
$ cd ~/Downloads/mediapipe/
# workon mediapipe
$ (mediapipe) pip3 install -r requirements.txt
$ (mediapipe) python3 -m pip install cython
$ (mediapipe) python3 -m pip install pillow
$ (mediapipe) sudo python3 setup.py gen_protos && python3 setup.py bdist_wheel
```

12. Link cv2.so file from global environment to virtual environment:

    ln -s /usr/lib/python3.6/dist-packages/cv2/python-3.6/cv2.cpython-36m-aarch64-linux-gnu.so ~/.virtualenvs/mediapipe/lib/python3.6/site-packages/cv2.so
    

13. Finally, 'manually' install mediapipe:

    $ (mediapipe) cd ~/Downloads/mediapipe/dist/
    $ (mediapipe) python3 -m pip install mediapipe-0.8-cp36-cp36m-linux_aarch64.whl
    $ sudo apt-get install libcanberra-gtk-module
    
    $ (mediapipe) pip install enlighten pandas
    
    $ sudo jetson_clocks --fan
    $ sudo jetson_clocks --store
    

[efthimia145]

Thank you for your detailed answer! I will try it our and let you know!