Error while importing python package after building from source (ARM)

[alexdlukens]

Hello, I am trying to build Open3D on ARM and am encountering issues when using the pip package built by my system. When importing the pip package I receive an ImportError:

Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import open3d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/orin/.local/lib/python3.8/site-packages/open3d/__init__.py", line 90, in <module>
    from open3d.cpu.pybind import (camera, data, geometry, io, pipelines,
ImportError: /home/orin/.local/lib/python3.8/site-packages/open3d/cpu/pybind.cpython-38-aarch64-linux-gnu.so: cannot allocate memory in static TLS block

Platform: Nvidia Jetson AGX Orin Dev-Kit: OS: Ubuntu 20.04 with kernel: linux ubuntu 5.10.65-tegra Open3D version: 0.15.1 (commit: ed30e3b)

I have followed the guide in the documentation to install dependencies. Currently, I cannot get any built python package to install correctly, but my intent is to eventually build with CUDA support. The pre-built pip package installs and imports correctly.

Is there somewhere I can see the package versions that the ARM wheel was built with so I can use the same versions locally? I am surprised that the locally built python package does not work, because the Open3D GUI I build locally does work correctly.

Any help is appreciated Thanks!

[HSqure]

Hello, I am trying to build Open3D on ARM and am encountering issues when using the pip package built by my system. When importing the pip package I receive an ImportError:

Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import open3d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/orin/.local/lib/python3.8/site-packages/open3d/__init__.py", line 90, in <module>
    from open3d.cpu.pybind import (camera, data, geometry, io, pipelines,
ImportError: /home/orin/.local/lib/python3.8/site-packages/open3d/cpu/pybind.cpython-38-aarch64-linux-gnu.so: cannot allocate memory in static TLS block

Platform: Nvidia Jetson AGX Orin Dev-Kit: OS: Ubuntu 20.04 with kernel: linux ubuntu 5.10.65-tegra Open3D version: 0.15.1 (commit: ed30e3b)

I have followed the guide in the documentation to install dependencies. Currently, I cannot get any built python package to install correctly, but my intent is to eventually build with CUDA support. The pre-built pip package installs and imports correctly.

Is there somewhere I can see the package versions that the ARM wheel was built with so I can use the same versions locally? I am surprised that the locally built python package does not work, because the Open3D GUI I build locally does work correctly.

Any help is appreciated Thanks!

Problem

Hi @alexdlukens , I met the similar problem at the installation Stepmake install-pip-package:

...

[100%] Generating python ops.py and return_types.py
Namespace(input_ops_py_in='/home/alinx/Open3D/python/open3d/ml/torch/python/ops.py.in', input_return_types_py_in='/home/alinx/Open3D/python/open3d/ml/torch/python/return_types.py.in', lib='/home/alinx/Open3D/build/lib/Release/cuda/open3d_torch_ops.so', output_dir='/home/alinx/Open3D/build/lib/ml/torch/python/', tensorflow_ops_dir='/home/alinx/Open3D/cpp/pybind/../open3d/ml/tensorflow')
Traceback (most recent call last):
  File "generate_torch_ops_wrapper.py", line 220, in <module>
    sys.exit(main())
  File "generate_torch_ops_wrapper.py", line 156, in main
    torch.ops.load_library(args.lib)
  File "/home/alinx/.virtualenvs/pt/lib/python3.8/site-packages/torch/_ops.py", line 220, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/alinx/Open3D/build/lib/Release/cuda/open3d_torch_ops.so: cannot allocate memory in static TLS block
make[3]: *** [cpp/pybind/CMakeFiles/python-package.dir/build.make:75: lib/ml/torch/python/ops.py] Error 1
make[2]: *** [CMakeFiles/Makefile2:3590: cpp/pybind/CMakeFiles/python-package.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:3623: cpp/pybind/CMakeFiles/pip-package.dir/rule] Error 2
make: *** [Makefile:1099: pip-package] Error 2

Solution

But after I try the following command before running make install-pip-package, It seems like be solved:

export LD_PRELOAD=/home/alinx/Open3D/build/lib/Release/cuda/open3d_torch_ops.so

Where LD_PRELOAD is the PATH of ERROR dynamic library file xxx.so.

In your situation, you can try:

export LD_PRELOAD=/home/orin/.local/lib/python3.8/site-packages/open3d/cpu/pybind.cpython-38-aarch64-linux-gnu.so

I think It may work, good luck!

Conference:

ImportError: dlopen: cannot load any more object with static TLS #2575

[pratt3000]

I am facing the same error on apple M1 when trying to import open3d. I am on python 3.9 using conda. Did you manage to resolve it @alexdlukens ?

[alexdlukens]

@pratt3000 No, this was a breaking issue for us. We returned to an x86 platform where we encountered less issues.

I imagine on Apple M1 you will not be able to use the GPU acceleration regardless. Are you using the prebuilt Python package? If so I would raise another GitHub issue for that

[bjajoh]

All above solutions don't work for me. I compiled it in a ubuntu 20.04 arm docker container on the M1.

[qscacheri]

@bjajoh Any tips for this? I'm trying to do the exact same thing, ubuntu container on m1 and am still getting the error.

[pratt3000]

Just downgrade your python. I got it after playing around with Python versions

[JunLeecus]

Same problem on NVIDIA Orin NX (with jetpack5.1) with BUILD_CUDA_MODULE=ON, while using pre-built cpu version works fine. And package built with same options on NVIDIA Jetson NX(with jetpack4.6.1) can work well.

[manbehindthemadness]

Identical problem on Nvidia AGX Orin:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/lib/python3.10/site-packages/open3d/__init__.py", line 93, in <module>
    from open3d.cpu.pybind import (core, camera, data, geometry, io, pipelines,
ImportError: /opt/venv/lib/python3.10/site-packages/open3d/cpu/pybind.cpython-310-aarch64-linux-gnu.so: cannot allocate memory in static TLS block

Using the LD_PRELOAD method causes python to core dump on load. Linux artax 5.10.104-tegra #1 SMP PREEMPT Sun Mar 19 07:55:28 PDT 2023 aarch64 aarch64 aarch64 GNU/Linux JetPack 5.1.1 L4T 35.3.1 Python 3.10.6

[manbehindthemadness]

Looking deeper it would appear the build having to be performed on CPU and GPU in separate alternation, they both are attempting to allocate the same memory block assuming they are in two different locations; however, in a Tegra architecture the memory for the CPU and GPU is the same physical silicon.

[jackpittenger]

Hello, I will add on my report of the similar issue. ImportWarning: Open3D was built with CUDA support, but an error ocurred while loading the Open3D CUDA Python bindings. This is usually because the CUDA libraries could not be found. Check your CUDA installation. Falling back to the CPU pybind library. Reported error: <>/pybind.cpython-38-aarch64-linux-gnu.so: cannot allocate memory in static TLS block. Python3.8, ARM Jetson, CUDA enabled. LD_PRELOAD did not work.

Anyone get this working?

[manbehindthemadness]

Is that a prebuilt package?On Jun 12, 2023, at 12:50 AM, Jack @.***> wrote: Hello, I will add on my report of the similar issue. ImportWarning: Open3D was built with CUDA support, but an error ocurred while loading the Open3D CUDA Python bindings. This is usually because the CUDA libraries could not be found. Check your CUDA installation. Falling back to the CPU pybind library. Reported error: <>/pybind.cpython-38-aarch64-linux-gnu.so: cannot allocate memory in static TLS block. Python3.8, ARM Jetson, CUDA enabled. LD_PRELOAD did not work. Anyone get this working?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[jackpittenger]

Is that a prebuilt package?On Jun 12, 2023, at 12:50 AM, Jack @.***> wrote: Hello, I will add on my report of the similar issue. ImportWarning: Open3D was built with CUDA support, but an error ocurred while loading the Open3D CUDA Python bindings. This is usually because the CUDA libraries could not be found. Check your CUDA installation. Falling back to the CPU pybind library. Reported error: <>/pybind.cpython-38-aarch64-linux-gnu.so: cannot allocate memory in static TLS block. Python3.8, ARM Jetson, CUDA enabled. LD_PRELOAD did not work. Anyone get this working?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

No, it is built from source with cuda and gui enabled

[manbehindthemadness]

Try adding /usr/local/cuda into the path variable 

[jackpittenger]

Try adding /usr/local/cuda into the path variable

It was already in the path. /usr/local/cuda & /usr/local/cuda/bin are in my path. Still gives the above runtime failure

[manbehindthemadness]

Hopefully one of the devs can shed some light onto this matter

[AwokeKnowing]

I also spent a long time setting up and building on Orin NX 16GB only for it to fail. As numpy and pytorch work fine, I expect there will be a rather small patch needed to deal with the unified memory issue which seems to be the core problem.

[manbehindthemadness]

I hear you, this is putting an unfortunate delay on my SLAM project, investigating switching to Kimera

[rty813]

Is there any progress now?

[rty813]

I temporarily resolved this issue. I enabled the BUILD_SHARED_LIBS=ON compile option on NVIDIA Orin NX (with jetpack5.1), and after running make, a libOpen3D.so file was generated. By setting LD_PRELOAD to the absolute path of this library file, the error no longer occurs.

Tips: The path tolibOpen3D.so is different for each person. You can check the path to the pybind shared library linked using 'ldd,' as shown in the following image: