Issue with `devices` flag on multi-GPU system

[GregoryKimball]

Recently when benchmarking libcudf on a DGX system, I ran into an issue where the MR setup by libcudf would only be respected by nvbench on GPU0. We observed that a CUDA MR would be used in place of the MR provided by libcudf. However, the compute did run on the correct GPU as specified by devices, so the root cause may be different than some related issues (e.g. https://github.com/NVIDIA/nvbench/pull/113).

This works and uses the pool MR default on GPU4 nsys profile -f true --gpu-metrics-device=all --output=report_cudavis --env-var CUDA_VISIBLE_DEVICES=4 ./STREAM_COMPACTION_NVBENCH -d 0 -b 1 -a NumRows=100000000 --timeout 0.3 -a Type=[I32] -a keep=[any] -a cardinality=10000000

This does not work and somehow uses a CUDA MR on GPU4 nsys profile -f true --gpu-metrics-device=all --output=report ./STREAM_COMPACTION_NVBENCH -d 4 -b 1 -a NumRows=100000000 --timeout 0.3 -a Type=[I32] -a keep=[any] -a cardinality=10000000

20241018_report.zip

[alliepiper]

@GregoryKimball I suspect that this was fixed by #113 (merged today), which fixes a bug where the launch.get_stream() stream may be associated with an incorrect device. Can you check whether this has fixed your issue?

[alliepiper]

Assuming #113 fixed this. Reopen if not.