Missing GPU kernels when using @profile and -b flag

[dwchang79]

I am using the @profile and -b flag to try and remove the initial training section of a ML workload so that I can only profile the inference part. That is working, but the problem is the GPU kernels and information are now missing. The call stack shows the functions, but they do not link to the GPU and no GPU devices are shown nor is anything shown running on them.

I have attached two screenshots. One with the entire run (without the profile flags) where the GPU section is shown at the bottom as "HIP Activity Device 2, Queue 0" and a second screenshot where only the inference part is profiled, but the GPU information is now gone.

Thank you.

[jrmadsen]

Try prefixing the command with omnitrace-run -- python3 -m omnitrace -b -- <script> <script-args>. I suspect the later initialization of omnitrace due to the @profile is causing in omnitrace getting initialized after the hip runtime, resulting in omnitrace not getting registered as profiling tool for the HIP runtime.

[ppanchad-amd]

Hi @dwchang79. Has your issue been resolved? If so, please close the ticket. Thanks!

[ppanchad-amd]

Hi @dwchang79. Closing ticked due to lack of activity. Please feel free to re-open ticket if you still see the issue with the latest ROCm. Thanks!