ROCm-Developer-Tools
Core dumped with performance counter mode profiling
I am experiencing a core dumped error when I am running a simple HIP program with the profiler.
I am running the vector add example from the HIP-Examples repo. After making the executable file. I used /opt/rocm/bin/rocprof --stats vectoradd_hip.exe to profile the program. The program cannot properly terminate and I see this error:
/opt/rocm/bin/rocprof: line 275: 17919 Aborted (core dumped) "vectoradd_hip.exe"
Here is my machine configuration:
ROCm version: 3.5
"name -a":
Linux lowfreq 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
"lsb_release -a":
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
"rocm-info":
[37mROCk module is loaded[0m
[37mAble to open /dev/kfd read-write[0m
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
==========
HSA Agents
==========
*******
Agent 1
*******
Name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Uuid: CPU-XX
Marketing Name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3100
BDFID: 0
Internal Node ID: 0
Compute Unit: 20
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 65878732(0x3ed3acc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 65878732(0x3ed3acc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
N/A
*******
Agent 2
*******
Name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Uuid: CPU-XX
Marketing Name: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 1
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3100
BDFID: 0
Internal Node ID: 1
Compute Unit: 20
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 66033596(0x3ef97bc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 66033596(0x3ef97bc) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
N/A
*******
Agent 3
*******
Name: gfx803
Uuid: GPU-XX
Marketing Name: Fiji [Radeon R9 FURY / NANO Series]
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 4096(0x1000)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 2
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
Chip ID: 29440(0x7300)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1000
BDFID: 33280
Internal Node ID: 2
Compute Unit: 64
SIMDs per CU: 4
Shader Engines: 4
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: FALSE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 4194304(0x400000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx803
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*******
Agent 4
*******
Name: gfx803
Uuid: GPU-XX
Marketing Name: Fiji [Radeon R9 FURY / NANO Series]
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 4096(0x1000)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 3
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
Chip ID: 29440(0x7300)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1000
BDFID: 33536
Internal Node ID: 3
Compute Unit: 64
SIMDs per CU: 4
Shader Engines: 4
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Features: KERNEL_DISPATCH
Fast F16 Operation: FALSE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 4194304(0x400000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx803
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
@eshcherb Any update on this thread? I recently noticed that even simply running rocprof --list-basic can cause the same problem.
Abort (core dumped) is exit of profiler on an error. Could you provide complete log with the error message?
I'm seeing the same error when trying to collect perf counters via the CLI tool (rocprof) and also while just trying to list the counters. ROCm 4.0 and the issue is the same. When I'm reading counters from an app that doesn't launch kernels (rocminfo for eg.) it doesn't crash, so at least init and cleanup aren't at fault.
@syifan are you looking for the actual dump file?
Hi, it looks like you have corrupted ROCM installation. ROCr runtime cannot find 'libhsa-amd-aqlprofile64.so' library.
Run the profiling with 'LD_DEBUG=libs':
$ LD_DEBUG=libs rocprof
And you can check where it looks 'libhsa-amd-aqlprofile64.so' library. It should be there '/opt/rocm/lib/libhsa-amd-aqlprofile64.so'. If you have it then it might that you didn't update the ROCM properly and you have an old version with incompatible API.
So I would recommend to wipe out your current ROCM completely and install the latest according the instructions from ROCM GitHub page: https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html
@eshcherb It was down to libhsa-amd-aqlprofile64.so missing from the install. Issuing sudo apt install --reinstall hsa-amd-aqlprofile did the trick. Thank you for the pointer.
Even though I work for a company subcontracted by AMD and am fairly well versed in the ways of administering a ROCm install, even I don't know where to report issues regarding the "bigger picture" of installing ROCm libraries. Few things to note:
- When looking for the package that provides said missing library, common APT tools can't fetch file information from the repository. If I'm not mistaken, it's
Contents.gzmissing from the dist, but I may be mistaken here. Something along the lines of this.
PS /home/mate> apt search aqlprofile
Sorting... Done
Full Text Search... Done
hsa-amd-aqlprofile/Ubuntu 16.04,now 1.0.0 amd64 [installed]
AQLPROFILE library for AMD HSA runtime API extension support
hsa-amd-aqlprofile4.0.0/Ubuntu 16.04 1.0.0 amd64
AQLPROFILE library for AMD HSA runtime API extension support
PS /home/mate> apt-file show hsa-amd-aqlprofile
PS /home/mate> apt-file show hsa-amd-aqlprofile4.0.0
Downloading the actual deb file and listing it's contents show that indeed this is the pacakge.
dpkg --contents ./hsa-amd-aqlprofile_1.0.0_amd64.deb
drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/
drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/
drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/
drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/
lrwxrwxrwx root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so -> libhsa-amd-aqlprofile64.so.1
lrwxrwxrwx root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so.1 -> libhsa-amd-aqlprofile64.so.1.0.40000
-rw-r--r-- root/root 249304 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so.1.0.40000
drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/lib/
lrwxrwxrwx root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/lib/libhsa-amd-aqlprofile64.so -> ../hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so
- While it is the recommended way of always doing a clean install of ROCm, it's not how users of APT repos have been socialized. Spamming
apt upgradeand things breaking is generally bad experience. I'm fairly certain nobody went and started deleting libraries inside a ROCm install. The package maintainers should really up their game in terms of test coverage of system upgrades. I'm not sure what's inside the current battery of deployment testing, but changes to the repo contents should likely test upgrade paths from the past 1-2 versions inside docker containers, VMs or anything.
I have been upgrading to ROCm 4.0 and the problem is not there anymore. So I guess @MathiasMagnus is right, it should be an installation issue.
