"build.sh" does not automatically find GPU on a Jetson and compiles for all architectures [BUG]

[znmeb]

Steps/Code to reproduce bug Follow the instructions on this repo's README. When you get to the "build.sh" step, it will do this:

➤ ./build.sh                                                                                                                                                          (base)
Building for the architecture of the GPU in the system...
Unable to determine GPU(s) installed...
Building for *ALL* supported GPU architectures...
         including: CUDA 10.X - {50,52,53,60,61,62,70,72,75}

Expected behavior I expected it to see I was on an AGX-Xavier and only build for that GPU.

Environment details:

• Environment location: AGX Xavier, both on the device itself and in the NGC "ML" container
• Method of cuSignal install: conda

Additional context I looked at the script and it's doing this:

NUMGPU=`lspci | grep VGA | grep NVIDIA | wc -l`

On the AGX Xavier, lspci only returns this:

➤ lspci                                                                                                                                                               (base)
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)

[awthomp]

Thanks for reporting this, @znmeb. I'll take a look over this week. The Jetson Xavier is CC 72, so it should be working.

[znmeb]

Yeah, it works. Since I went ahead and set the environment variable to specify the GPU for the cupy build, I figured you could accept the same variable building cusignal - leave it up to the user on Jetsons, since it works fine on x86_64 as is.

[awthomp]

Yeah, it works. Since I went ahead and set the environment variable to specify the GPU for the cupy build, I figured you could accept the same variable building cusignal - leave it up to the user on Jetsons, since it works fine on x86_64 as is.

I'm confused. If you follow the build instructions listed on the main README for cuSignal, from source, on a Jetson, ./build.sh does or doesn't error out? Are you just saying that the build script doesn't recognize the CC 72 and therefore builds for all compute architectures? In other words, cuSignal builds successfully, but the build process is too long?

Thanks!

[znmeb]

Sorry ... I miswrote the bug report. cusignal compiles for all GPUs and executes correctly on the Jetson ... it just doesn't detect the GPU so it builds all of the versions.

[awthomp]

Thanks for the clarification! We'll take a look :).

[mnicely]

@znmeb Do you mind setting export CUDA_VISIBLE_DEVICES=0 and rerunning build.sh?

[github-actions[bot]]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions[bot]]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[evanmayer]

I ran into this issue too. @awthomp's suggestion of export CUDA_VISIBLE_DEVICES=0 yields

Building for the architecture of the GPU in the system...
        Device 0 - CC 53

which seems like the intended behavior when following the Jetson build instructions.

[awthomp]

Thanks for confirming, @evanmayer. I'm going to close this issue as it seems there's an acceptable work-around.