Tracing cuda functions

[sujankh]

Hi, Is it possible to trace functions in CUDA runtime (libcudart.so) like cudaMalloc? I tried this $ sudo trace-bpfcc 'p:cudart:cudaMalloc "size = %d", arg2', it runs but it does not intercept any cudaMalloc calls from my cuda application. I tried searching around but did not have any luck. Thank you.

[chenhengqi]

Run ldd on your cuda application to find out which libcudart.so you are linked against, and use the -s option to supply that so for symbol resolution.

[zxgm]

I encountered the same problem when use sudo python3 /usr/sbin/trace-bpfcc 'p:cudart:cudaMalloc "size = %d", arg2' I want to tracing tensorflow or pytorch gpu training cuda api that coded by python

[zxgm]

I encountered the same problem when use sudo python3 /usr/sbin/trace-bpfcc 'p:cudart:cudaMalloc "size = %d", arg2' I want to tracing tensorflow or pytorch gpu training cuda api that coded by python

The problem already solved, because pytorch runs with the built-in cuda dynamic library instead of the library under the cuda installation path

[sujankh]

Forgot to update here last year: I realized that my binary was statically linked.

I solved my issue by specifying the path to my binary like this:

trace-bpfcc 'p:/path/to/my-binary:cudaMalloc'

and this worked