RadeonOpenCompute
[Issue]: Memory copy started by dependency signal for pinned memory exhibits race condition
Problem Description
Scheduling a memory copy with hsa_amd_memory_async_copy and using a dependency signal to start the copy after hsaKmtMapMemoryToGPUNodes has returned seems to result in a race condition. See the attached tarball. Reproduction instructions and other information is in README.md. The exhibiting program is in main.cpp.
Operating System
Archlinux
CPU
AMD Ryzen Threadripper 1950X
GPU
AMD Radeon Pro W6800
ROCm Version
ROCm 6.2.2
ROCm Component
ROCK-Kernel-Driver
Steps to Reproduce
Unzip the attached tarball, and follow the instructions in README.md.
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES
==========
HSA Agents
==========
*******
Agent 1
*******
Name: AMD Ryzen Threadripper 1950X 16-Core Processor
Uuid: CPU-XX
Marketing Name: AMD Ryzen Threadripper 1950X 16-Core Processor
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 3400
BDFID: 0
Internal Node ID: 0
Compute Unit: 32
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 131770752(0x7daa980) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 131770752(0x7daa980) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 131770752(0x7daa980) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 2
*******
Name: gfx1030
Uuid: GPU-c5f17292dba09909
Marketing Name: AMD Radeon Pro W6800
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
L2: 4096(0x1000) KB
L3: 131072(0x20000) KB
Chip ID: 29603(0x73a3)
ASIC Revision: 1(0x1)
Cacheline Size: 128(0x80)
Max Clock Freq. (MHz): 2555
BDFID: 3328
Internal Node ID: 1
Compute Unit: 60
SIMDs per CU: 2
Shader Engines: 4
Shader Arrs. per Eng.: 2
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 32(0x20)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 32(0x20)
Max Work-item Per CU: 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 118
SDMA engine uCode:: 83
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 31440896(0x1dfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 31440896(0x1dfc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx1030
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
Additional Information
No response
Just a thought that occurred to me: I don't think the bug is specific to the SDMA. It's just more easily reproduced that way. If I put a second GPU in the machine, it seems to take longer for the GPU to see the pinned memory - long enough to reproduce the problem with a kernel rather than SDMA copy.
Hi @nstomlinson. Internal ticket has been created to investigate your issue. Thanks!
Hi @nstomlinson, I couldn't repro the issue with any/no wait time. Can you send the core dump?
Hello
- I've attached a core dump, binaries with debug information, and new system information.
- I ran the build and test in the Docker image for Ubuntu 24.04 this time to get a core dump that better matches what you might expect.
- I did notice it took several attempts to get the reproducer to crash.
- It should be possible to catch it crashing with
for I in {0..99} ; do timeout 10 ./reproducer 0 ; if [ "$?" != "124" ] ; then break ; fi ; done(though in practice, a shorter timeout and fewer iterations should suffice). - The system is also a bit different this time, in particular, there are now two GPUs in this machine. The tarball includes
lspciandrocminfofor this reason.
Thanks!
Thanks for the information. I can now reproduce the error with the libs that you provided. I'll look into it. Can you tell me how were the .so files built? I also noticed that you have
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
With ROCm 6.2.2 it should look like:
ROCk module version 6.8.5 is loaded
=====================
HSA System Attributes
=====================
Runtime Version: 1.14
Runtime Ext Version: 1.6
I suspect that it's a problem in the kernel driver. Since you are on Arch which is not officially supported, kfd may have unexpected behaviours. Since you need to expose kfd to the docker container, I believe that's also why it fails on Ubuntu. Can you check if the problem if reproducible locally on Ubuntu?
I ran rocminfo from the host machine, which is on the ROCm 6.0.2 userspace. That probably accounts for the difference in HSA system attributes you observed. There's a build.sh script in the original tarball I attached. I built the libraries I provided like ./build.sh Debug.
Can you update ROCm on the host to 6.2.2 and check if the problem persists?
I did (by building the parts of ROCm userspace that it uses). It does still happen in that case. I haven't tested bare metal Ubuntu yet though.
You can also try 6.2.1 via AUR, see https://aur.archlinux.org/packages/opencl-amd. Unfortunately, Arch is not officially supported so I recommend you use any of the supported OS listed here if possible.
Closing due to lack of recent activity. Please update the issue if it persists and I'll reopen it.
Closing due to lack of recent activity. Please update the issue if it persists and I'll reopen it.
Has a kernel patch been submitted etc., or was this issue just closed due to lack of activity?
If no action has been taken to fix it, then it does indeed persist!
A reproducer has been provided in the initial report. Does this work for you?
Hi @nstomlinson @powderluv. We tested on the latest system package (ROCm 6.2.4) and ROCm 6.2.2 (the reported version) on Arch and after a full system package update (which updates everything including the kernel to the latest versions in Arch), the issue is no longer reproducible. It appears the issue has been fixed somewhere in the system package (+200 packages+kernel). Thanks!
Hi @nstomlinson. Closing ticket. Please re-open if issue persists. Thanks!