omnitrace
omnitrace copied to clipboard
ROCm
`omnitrace-avail` fails on ROCM 5.3 and RX 6800XT
Using the following docker container:
FROM rocm/rocm-terminal:5.3
RUN sudo apt-get update && \
sudo apt-get dist-upgrade -y && \
sudo apt-get install -y wget vim
RUN wget https://github.com/AMDResearch/omnitrace/releases/download/v1.10.0/omnitrace_1.10.0-ubuntu20.04.ROCm50300.PAPI.OMPT.Python3_amd64.deb && \
sudo dpkg -i omnitrace_1.10.0-ubuntu20.04.ROCm50300.PAPI.OMPT.Python3_amd64.deb && \
rm *.deb
Running with: docker run -it --device=/dev/kfd --device=/dev/dri --group-add video
*******
Agent 2
*******
Name: gfx1030
Uuid: GPU-849e7f8e4045a584
Marketing Name: AMD Radeon RX 6800 XT
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
rocm-user@0010d1870e71:~$ /opt/omnitrace/bin/omnitrace-avail
[omnitrace][184][0][fatal]
[omnitrace][184][0][fatal] ERROR :: rocprofiler_iterate_info(), ImportMetrics(), Bad metric 'L2CacheHit', var 'TCC_HIT[0]' is not found
[omnitrace][184][0] Signal 6 caught : Aborted (Signal sent by tkill() 184 1000)
### ERROR ### [omnitrace][PID=184][TID=0] signal=6 (SIGABRT) abort program (formerly SIGIOT). code: -6
Backtrace:
[PID=184][TID=0][0/9] __restore_rt
[PID=184][TID=0][1/9] gsignal +0xcb
[PID=184][TID=0][2/9] abort +0x12b
[PID=184][TID=0][3/9] _ZN9omnitrace11rocprofiler5fatalERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE.cold +0x5
[PID=184][TID=0][4/9] _ZN9omnitrace11rocprofiler17rocm_check_statusE12hsa_status_tRKSt3setIS1_St4lessIS1_ESaIS1_EE +0xa0
[PID=184][TID=0][5/9] _ZN9omnitrace11rocprofiler12rocm_metricsEv +0x156
[PID=184][TID=0][6/9] main +0x3d47
[PID=184][TID=0][7/9] __libc_start_main +0xf3
[PID=184][TID=0][8/9] _start +0x2e
Backtrace (demangled):
[PID=184][TID=0][0/9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f02b03f1420]
[PID=184][TID=0][1/9] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f02ae30200b]
[PID=184][TID=0][2/9] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f02ae2e1859]
[PID=184][TID=0][3/9] /opt/omnitrace/bin/omnitrace-avail(+0x129aab) [0x55b1fe3d5aab]
[PID=184][TID=0][4/9] /opt/omnitrace/bin/omnitrace-avail(+0x6f99b0) [0x55b1fe9a59b0]
[PID=184][TID=0][5/9] /opt/omnitrace/bin/omnitrace-avail(+0x6fac16) [0x55b1fe9a6c16]
[PID=184][TID=0][6/9] /opt/omnitrace/bin/omnitrace-avail(+0x4b87b7) [0x55b1fe7647b7]
[PID=184][TID=0][7/9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f02ae2e3083]
[PID=184][TID=0][8/9] /opt/omnitrace/bin/omnitrace-avail(+0x4f3b8e) [0x55b1fe79fb8e]
/proc/184/maps:
55b1fe2ac000-55b1fe3cb000 r--p 00000000 00:18 40421 /opt/omnitrace/bin/omnitrace-avail
55b1fe3cb000-55b20030f000 r-xp 0011f000 00:18 40421 /opt/omnitrace/bin/omnitrace-avail
55b20030f000-55b20086a000 r--p 02063000 00:18 40421 /opt/omnitrace/bin/omnitrace-avail
55b20086b000-55b2008b2000 r--p 025be000 00:18 40421 /opt/omnitrace/bin/omnitrace-avail
55b2008b2000-55b20092b000 rw-p 02605000 00:18 40421 /opt/omnitrace/bin/omnitrace-avail
55b20092b000-55b200f5d000 rw-p 00000000 00:00 0
55b201875000-55b20250a000 rw-p 00000000 00:00 0 [heap]
7f0197a00000-7f0198241000 r--p 00000000 00:18 4529 /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
7f0198241000-7f019d738000 r-xp 00841000 00:18 4529 /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
7f019d738000-7f019f908000 r--p 05d38000 00:18 4529 /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
7f019f908000-7f019f909000 ---p 07f08000 00:18 4529 /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
7f019f909000-7f019fe4c000 r--p 07f08000 00:18 4529 /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
7f019fe4c000-7f019fe9c000 rw-p 0844b000 00:18 4529 /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
7f019fe9c000-7f019fefc000 rw-p 00000000 00:00 0
7f01a0000000-7f01a0021000 rw-p 00000000 00:00 0
7f01a0021000-7f01a4000000 ---p 00000000 00:00 0
7f01a5800000-7f02a5800000 ---p 00000000 00:00 0
7f02a6cae000-7f02a8000000 rw-p 00000000 00:00 0
7f02a8000000-7f02a8021000 rw-p 00000000 00:00 0
7f02a8021000-7f02ac000000 ---p 00000000 00:00 0
7f02ac59c000-7f02acb76000 rw-p 00000000 00:00 0
7f02acb76000-7f02acc07000 rw-p 00000000 00:00 0
7f02acc67000-7f02acca7000 rw-p 00000000 00:00 0
7f02acca7000-7f02accb1000 r--p 00000000 00:18 4542 /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
7f02accb1000-7f02acce4000 r-xp 0000a000 00:18 4542 /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
7f02acce4000-7f02accef000 r--p 0003d000 00:18 4542 /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
7f02accef000-7f02accf0000 ---p 00048000 00:18 4542 /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
7f02accf0000-7f02accf3000 r--p 00048000 00:18 4542 /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
7f02accf3000-7f02acd00000 rw-p 0004b000 00:18 4542 /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
7f02acd00000-7f02ace01000 rw-p 00000000 00:00 0
7f02ace09000-7f02ace3a000 rw-p 00000000 00:00 0
7f02ace6b000-7f02acf00000 rw-p 00000000 00:00 0
7f02acf00000-7f02ad001000 rw-p 00000000 00:00 0
7f02ad015000-7f02ad05b000 rw-p 00000000 00:00 0
7f02ad05b000-7f02ad069000 r--p 00000000 00:18 1307 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f02ad069000-7f02ad078000 r-xp 0000e000 00:18 1307 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f02ad078000-7f02ad086000 r--p 0001d000 00:18 1307 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f02ad086000-7f02ad08a000 r--p 0002a000 00:18 1307 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f02ad08a000-7f02ad08b000 rw-p 0002e000 00:18 1307 /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
7f02ad08b000-7f02ad08c000 ---p 00000000 00:00 0
7f02ad08c000-7f02ad88c000 rwxp 00000000 00:00 0
7f02ad88c000-7f02ad88d000 ---p 00000000 00:00 0
7f02ad88d000-7f02ae08d000 rwxp 00000000 00:00 0
7f02ae08d000-7f02ae0dd000 rw-p 00000000 00:00 0
7f02ae0dd000-7f02ae173000 r--p 00000000 00:18 1297 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f02ae173000-7f02ae264000 r-xp 00096000 00:18 1297 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f02ae264000-7f02ae2ad000 r--p 00187000 00:18 1297 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f02ae2ad000-7f02ae2ae000 ---p 001d0000 00:18 1297 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f02ae2ae000-7f02ae2b9000 r--p 001d0000 00:18 1297 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f02ae2b9000-7f02ae2bc000 rw-p 001db000 00:18 1297 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f02ae2bc000-7f02ae2bf000 rw-p 00000000 00:00 0
7f02ae2bf000-7f02ae2e1000 r--p 00000000 00:18 1187 /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f02ae2e1000-7f02ae459000 r-xp 00022000 00:18 1187 /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f02ae459000-7f02ae4a7000 r--p 0019a000 00:18 1187 /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f02ae4a7000-7f02ae4ab000 r--p 001e7000 00:18 1187 /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f02ae4ab000-7f02ae4ad000 rw-p 001eb000 00:18 1187 /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f02ae4ad000-7f02ae4b1000 rw-p 00000000 00:00 0
7f02ae4b1000-7f02ae4be000 r--p 00000000 00:18 1229 /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f02ae4be000-7f02ae565000 r-xp 0000d000 00:18 1229 /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f02ae565000-7f02ae5fe000 r--p 000b4000 00:18 1229 /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f02ae5fe000-7f02ae5ff000 r--p 0014c000 00:18 1229 /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f02ae5ff000-7f02ae600000 rw-p 0014d000 00:18 1229 /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f02ae600000-7f02ae61e000 r--p 00000000 00:18 4532 /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
7f02ae61e000-7f02ae99e000 r-xp 0001e000 00:18 4532 /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
7f02ae99e000-7f02aea8e000 r--p 0039e000 00:18 4532 /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
7f02aea8e000-7f02aea8f000 ---p 0048e000 00:18 4532 /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
7f02aea8f000-7f02aea95000 r--p 0048e000 00:18 4532 /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
7f02aea95000-7f02affc9000 rw-p 00494000 00:18 4532 /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
7f02affc9000-7f02affda000 rw-p 00000000 00:00 0
7f02affe6000-7f02affe7000 rw-p 00000000 00:00 0
7f02affe8000-7f02afff0000 rw-s 107ae6000 00:b8 15 /dev/dri/renderD128
7f02afff2000-7f02afff4000 rw-p 00000000 00:00 0
7f02afff5000-7f02afff6000 rw-p 00000000 00:00 0
7f02afff7000-7f02b0000000 rw-p 00000000 00:00 0
7f02b0000000-7f02b00ef000 r--p 00000000 00:18 40673 /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
7f02b00ef000-7f02b0100000 r-xp 000ef000 00:18 40673 /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
7f02b0100000-7f02b0193000 r--p 00100000 00:18 40673 /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
7f02b0193000-7f02b0194000 ---p 00193000 00:18 40673 /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
7f02b0194000-7f02b023c000 r--p 00193000 00:18 40673 /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
7f02b023c000-7f02b029d000 rw-p 0023b000 00:18 40673 /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
7f02b029d000-7f02b029f000 rw-p 00000000 00:00 0
7f02b029f000-7f02b02a4000 rw-p 00000000 00:00 0
7f02b02a4000-7f02b02a7000 r--p 00000000 00:18 19012 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f02b02a7000-7f02b02ac000 r-xp 00003000 00:18 19012 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f02b02ac000-7f02b02ae000 r--p 00008000 00:18 19012 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f02b02ae000-7f02b02af000 ---p 0000a000 00:18 19012 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f02b02af000-7f02b02b0000 r--p 0000a000 00:18 19012 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f02b02b0000-7f02b02b1000 rw-p 0000b000 00:18 19012 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
7f02b02b1000-7f02b02b4000 r--p 00000000 00:18 3907 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
7f02b02b4000-7f02b02bb000 r-xp 00003000 00:18 3907 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
7f02b02bb000-7f02b02bd000 r--p 0000a000 00:18 3907 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
7f02b02bd000-7f02b02be000 r--p 0000b000 00:18 3907 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
7f02b02be000-7f02b02bf000 rw-p 0000c000 00:18 3907 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
7f02b02bf000-7f02b02c4000 r--p 00000000 00:18 3905 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
7f02b02c4000-7f02b02d1000 r-xp 00005000 00:18 3905 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
7f02b02d1000-7f02b02d5000 r--p 00012000 00:18 3905 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
7f02b02d5000-7f02b02d6000 r--p 00015000 00:18 3905 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
7f02b02d6000-7f02b02d7000 rw-p 00016000 00:18 3905 /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
7f02b02d7000-7f02b02da000 r--p 00000000 00:18 18872 /usr/lib/x86_64-linux-gnu/libelf-0.176.so
7f02b02da000-7f02b02ec000 r-xp 00003000 00:18 18872 /usr/lib/x86_64-linux-gnu/libelf-0.176.so
7f02b02ec000-7f02b02f1000 r--p 00015000 00:18 18872 /usr/lib/x86_64-linux-gnu/libelf-0.176.so
7f02b02f1000-7f02b02f2000 r--p 00019000 00:18 18872 /usr/lib/x86_64-linux-gnu/libelf-0.176.so
7f02b02f2000-7f02b02f3000 rw-p 0001a000 00:18 18872 /usr/lib/x86_64-linux-gnu/libelf-0.176.so
7f02b02f3000-7f02b02f5000 rw-p 00000000 00:00 0
7f02b02f5000-7f02b02f8000 r--p 00000000 00:18 1212 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f02b02f8000-7f02b030a000 r-xp 00003000 00:18 1212 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f02b030a000-7f02b030e000 r--p 00015000 00:18 1212 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f02b030e000-7f02b030f000 r--p 00018000 00:18 1212 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f02b030f000-7f02b0310000 rw-p 00019000 00:18 1212 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f02b0310000-7f02b032d000 r--p 00000000 00:18 4560 /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
7f02b032d000-7f02b038d000 r-xp 0001d000 00:18 4560 /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
7f02b038d000-7f02b03a2000 r--p 0007d000 00:18 4560 /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
7f02b03a2000-7f02b03a4000 r--p 00091000 00:18 4560 /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
7f02b03a4000-7f02b03a5000 rw-p 00093000 00:18 4560 /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
7f02b03a5000-7f02b03a7000 r--p 00000000 00:18 40701 /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
7f02b03a7000-7f02b03b2000 r-xp 00002000 00:18 40701 /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
7f02b03b2000-7f02b03b5000 r--p 0000d000 00:18 40701 /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
7f02b03b5000-7f02b03b6000 r--p 0000f000 00:18 40701 /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
7f02b03b6000-7f02b03b7000 rw-p 00010000 00:18 40701 /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
7f02b03b7000-7f02b03c1000 rw-p 00000000 00:00 0
7f02b03c1000-7f02b03c3000 r--p 00000000 00:18 33853 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f02b03c3000-7f02b03d4000 r-xp 00002000 00:18 33853 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f02b03d4000-7f02b03da000 r--p 00013000 00:18 33853 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f02b03da000-7f02b03db000 ---p 00019000 00:18 33853 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f02b03db000-7f02b03dc000 r--p 00019000 00:18 33853 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f02b03dc000-7f02b03dd000 rw-p 0001a000 00:18 33853 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
7f02b03dd000-7f02b03e3000 r--p 00000000 00:18 1281 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f02b03e3000-7f02b03f4000 r-xp 00006000 00:18 1281 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f02b03f4000-7f02b03fa000 r--p 00017000 00:18 1281 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f02b03fa000-7f02b03fb000 r--p 0001c000 00:18 1281 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f02b03fb000-7f02b03fc000 rw-p 0001d000 00:18 1281 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f02b03fc000-7f02b0400000 rw-p 00000000 00:00 0
7f02b0400000-7f02b0418000 r--p 00000000 00:18 4545 /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
7f02b0418000-7f02b04ef000 r-xp 00018000 00:18 4545 /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
7f02b04ef000-7f02b0558000 r--p 000ef000 00:18 4545 /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
7f02b0558000-7f02b0559000 ---p 00158000 00:18 4545 /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
7f02b0559000-7f02b0561000 r--p 00158000 00:18 4545 /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
7f02b0561000-7f02b069c000 rw-p 00160000 00:18 4545 /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
7f02b069c000-7f02b06a1000 rw-p 00000000 00:00 0
7f02b06a1000-7f02b06a2000 r--p 00000000 00:18 40657 /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
7f02b06a2000-7f02b06a8000 r-xp 00001000 00:18 40657 /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
7f02b06a8000-7f02b06ab000 r--p 00007000 00:18 40657 /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
7f02b06ab000-7f02b06ac000 r--p 00009000 00:18 40657 /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
7f02b06ac000-7f02b06ad000 rw-p 0000a000 00:18 40657 /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
7f02b06ad000-7f02b06af000 rw-p 00000000 00:00 0
7f02b06af000-7f02b06b1000 r--p 00000000 00:18 1285 /usr/lib/x86_64-linux-gnu/librt-2.31.so
7f02b06b1000-7f02b06b5000 r-xp 00002000 00:18 1285 /usr/lib/x86_64-linux-gnu/librt-2.31.so
7f02b06b5000-7f02b06b7000 r--p 00006000 00:18 1285 /usr/lib/x86_64-linux-gnu/librt-2.31.so
7f02b06b7000-7f02b06b8000 r--p 00007000 00:18 1285 /usr/lib/x86_64-linux-gnu/librt-2.31.so
7f02b06b8000-7f02b06b9000 rw-p 00008000 00:18 1285 /usr/lib/x86_64-linux-gnu/librt-2.31.so
7f02b06b9000-7f02b06ba000 r--p 00000000 00:18 1198 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f02b06ba000-7f02b06bc000 r-xp 00001000 00:18 1198 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f02b06bc000-7f02b06bd000 r--p 00003000 00:18 1198 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f02b06bd000-7f02b06be000 r--p 00003000 00:18 1198 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f02b06be000-7f02b06bf000 rw-p 00004000 00:18 1198 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f02b06bf000-7f02b06c6000 r--p 00000000 00:18 4563 /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
7f02b06c6000-7f02b0700000 r-xp 00007000 00:18 4563 /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
7f02b0700000-7f02b070d000 r--p 00041000 00:18 4563 /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
7f02b070d000-7f02b070e000 ---p 0004e000 00:18 4563 /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
7f02b070e000-7f02b070f000 r--p 0004e000 00:18 4563 /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
7f02b070f000-7f02b0710000 rw-p 0004f000 00:18 4563 /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
7f02b0710000-7f02b0715000 r--p 00000000 00:18 4566 /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
7f02b0715000-7f02b0742000 r-xp 00005000 00:18 4566 /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
7f02b0742000-7f02b0750000 r--p 00032000 00:18 4566 /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
7f02b0750000-7f02b0751000 r--p 0003f000 00:18 4566 /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
7f02b0751000-7f02b0752000 rw-p 00040000 00:18 4566 /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
7f02b0752000-7f02b0756000 rw-p 00000000 00:00 0
7f02b0756000-7f02b0757000 rw-s 00000000 00:bc 4 /dev/shm/rocm_smi_card0
7f02b0757000-7f02b0758000 rw-p 00000000 00:00 0
7f02b0758000-7f02b0759000 rw-p 00000000 00:00 0
7f02b0759000-7f02b075a000 rw-s 3e64800000000000 00:b8 11 /dev/kfd
7f02b075a000-7f02b075b000 rw-p 00000000 00:00 0
7f02b075b000-7f02b075c000 rw-s 00000000 00:bc 3 /dev/shm/hsakmt_shared_mem
7f02b075c000-7f02b075d000 r--p 00000000 00:18 1165 /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f02b075d000-7f02b0780000 r-xp 00001000 00:18 1165 /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f02b0780000-7f02b0788000 r--p 00024000 00:18 1165 /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f02b0788000-7f02b0789000 rw-s 00000000 00:bc 2 /dev/shm/sem.hsakmt_semaphore
7f02b0789000-7f02b078a000 r--p 0002c000 00:18 1165 /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f02b078a000-7f02b078b000 rw-p 0002d000 00:18 1165 /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f02b078b000-7f02b078c000 rw-p 00000000 00:00 0
7ffe321ac000-7ffe321ee000 rwxp 00000000 00:00 0 [stack]
7ffe321ee000-7ffe321f0000 rw-p 00000000 00:00 0
7ffe321f3000-7ffe321f7000 r--p 00000000 00:00 0 [vvar]
7ffe321f7000-7ffe321f9000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
Backtrace (demangled):
[PID=184][TID=0][0/9] __restore_rt
[PID=184][TID=0][1/9] gsignal +0xcb
[PID=184][TID=0][2/9] abort +0x12b
[PID=184][TID=0][3/9] omnitrace::rocprofiler::fatal(std::string const&) [clone .cold] +0x5
[PID=184][TID=0][4/9] omnitrace::rocprofiler::rocm_check_status(hsa_status_t, std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>> const&) +0xa0
[PID=184][TID=0][5/9] omnitrace::rocprofiler::rocm_metrics() +0x156
[PID=184][TID=0][6/9] main +0x3d47
[PID=184][TID=0][7/9] __libc_start_main +0xf3
[PID=184][TID=0][8/9] _start +0x2e
Backtrace (lineinfo):
[PID=184][TID=0][0/7]
[/lib/x86_64-linux-gnu/libpthread.so.0:?] __restore_rt
[PID=184][TID=0][1/7]
[/usr/lib/x86_64-linux-gnu/libc-2.31.so:?] abort
[PID=184][TID=0][2/7]
[/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:75] omnitrace::rocprofiler::fatal(std::string const&)
[PID=184][TID=0][3/7]
[/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:100] omnitrace::rocprofiler::rocm_check_status(hsa_status_t, std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>> const&)
[PID=184][TID=0][4/7]
[/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:370] omnitrace::rocprofiler::rocm_metrics()
[/usr/include/c++/9/bits/stl_set.h:281] std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>>::~set()
[/usr/include/c++/9/bits/stl_tree.h:1000] std::_Rb_tree<hsa_status_t, hsa_status_t, std::_Identity<hsa_status_t>, std::less<hsa_status_t>, std::allocator<hsa_status_t>>::~_Rb_tree()
[PID=184][TID=0][5/7]
[/home/omnitrace/source/bin/omnitrace-avail/avail.cpp:513] main
[/usr/include/c++/9/bits/stl_vector.h:916] std::vector<tim::hardware_counters::info, std::allocator<tim::hardware_counters::info>>::size() const
[PID=184][TID=0][6/7]
[/usr/lib/x86_64-linux-gnu/libc-2.31.so:?] __libc_start_main
[omnitrace][184] Finalizing afer signal 6 :: Signal: SIGABRT (signal number: 6) abort program (formerly SIGIOT)
[omnitrace][184] Killing process 184 with signal 6...
This works when switching to ROCM 5.4
FROM rocm/rocm-terminal:5.4
RUN sudo apt-get update && \
sudo apt-get dist-upgrade -y && \
sudo apt-get install -y wget vim
RUN wget https://github.com/AMDResearch/omnitrace/releases/download/v1.10.0/omnitrace_1.10.0-ubuntu20.04.ROCm50400.PAPI.OMPT.Python3_amd64.deb && \
sudo dpkg -i omnitrace_1.10.0-ubuntu20.04.ROCm50400.PAPI.OMPT.Python3_amd64.deb && \
rm *.deb
FYI, this is probably a rocprofiler bug, not an omnitrace bug. There have been a lot of issues with its stability between minor releases recently.
