nvtx in Python code [BUG]

[JasonAtNvidia]

Describe the bug There are several locations in the Python package of this repository that utilize the annotate feature of the nvtx Python package. NVTX is not available on all platforms (i.e. Jetson) and this also violates the flag --disable_nvtx when used in build.sh.

Steps/Code to reproduce bug Build cudf, try to import on platform without nvtx installed

Expected behavior Python skips nvtx when not available as a package and cudf continues like normal.

Environment overview (please complete the following information)

• Environment location: Jetson bare-metal
• Method of cuDF install: source

Environment details N/A

Additional context With Jetpack 5.0 released with CUDA 11.4 and Jetson Orin available with Ampere GPUs it should be easy to port over the long term support RAPIDS version 21.08 for Jetson and make the packages available.

[shwina]

Hi @JasonAtNvidia -- thanks for reporting. I'm curious if the steps in https://github.com/NVIDIA/NVTX/issues/43#issuecomment-924987584 help you install NVTX on Jetson?

[JasonAtNvidia]

@shwina Building the package specifically for Jetson did work.

git clone -b dev https://github.com/NVIDIA/NVTX.git
cd NVTX/python
C_INCLUDE_PATH=/usr/local/cuda-11.4/targets/aarch64-linux/include/ python3 setup.py bdist_conda

Should we have to do this considering the --disable_nvtx flag in build.sh? With NVTX coming out with Jetson CUDA 11.4 perhaps that flag is no longer needed?

[shwina]

With NVTX coming out with Jetson CUDA 11.4

Sorry - could you please clarify further here? NVTX - the C library - is a header-only library. Do you mean that Jetson CUDA 11.4 will include the NVTX headers?

[JasonAtNvidia]

So the original reason for the --disable_nvtx flag was a build for Jetson in 2020 when Jetson Xavier release was CUDA 10.2 but NVTX did not exist on Jetson yet. Jetpack 5.0.2 GA should release today and I just discovered they included NVTX headers with the CUDA 11.4 that comes with this release. But 2 things either need to happen, we need to get rid of the --disable_nvtx flag in build.sh and CMakeLists.txt or we need to isolate and flag around the Python library calls to nvtx. It does not make sense to disable nvtx in the C code but require it for Python.

[shwina]

build.sh can be passed flags for building the C++ source, but Python calls NVTX at runtime. I can't think of a great way to control that runtime behaviour via build.sh.

At runtime, you can make the nvtx.annotate() calls a no-op in Python via the environment variable DISABLE_NVTX=1, but I don't think that is relevant to your requirements or this discussion.

Taking a step back, I believe the underlying issue here is that the nvtx Python package is a dependency of cudf. In general, you should be able to install it via conda or pip. On Jetson, conda will work, while pip will not*. Thus, the only option is to build nvtx from source, using the steps I linked to above. Note that those steps don't need the NVTX headers to be present - they will be pulled from GitHub.

* the reason pip install nvtx doesn't work on Jetson is that we don't have ARM packages for NVTX up on PyPI. That is something I hope to fix very soon.

[JasonAtNvidia]

@shwina The nvtx from conda-forge didn't work on Jetson until I just built it specifically for Jetson just a few hours ago. It was failing originally, that is why I filed this bug. I'll have to include the specific build in the community repository, perhaps under a special l4t build flag.

[shwina]

The nvtx from conda-forge didn't work on Jetson until I just built it specifically for Jetson just a few hours ago. It was failing originally, that is why I filed this bug.

Can I ask how it was failing? What traceback did you get?

[shwina]

Hi @JasonAtNvidia -- we just pushed NVTX wheels for aarch64 on PyPI. I wonder if you're able to just pip install nvtx on Jetson now?

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[vyasr]

@shwina @JasonAtNvidia is this still an issue that needs addressing? If nvtx is now available on Jetson, perhaps the main action item at this point is to remove the --disable-nvtx flags and enforce that both C++ and Python always use NVTX?

[JasonAtNvidia]

@vyasr I think we can close this issue as it is no longer relevant. You are correct that nvtx libraries are universally available libraries now.