ROCR-Runtime
ROCR-Runtime copied to clipboard
RadeonOpenCompute
[Issue]: OpenCL fails on a Ryzen 5700G APU on ROCm 6.0.0 - regression (OpenCL buffer operations cause a ring sdma0 timeout & GPU reset)
Problem Description
After upgrading my Arch Linux system with a Ryzen 5700G APU to ROCm 6.0.0, running pretty much any OpenCL application (see below for an example) fails.
In particular, creating an OpenCL command queue, building a kernel, etc. works fine, but when trying to run operations over OpenCL buffers (i.e. clEnqueueReadBuffer, clEnqueueWriteBuffer), the application in the best scenario hangs (when running it on a TTY), and in the worst scenario causes a GPU reset (when running it on a graphical session). An error message *ERROR* ring sdma0 timeout is also logged to dmesg along with more logs related to the GPU reset (full logs included below).
If I downgrade ROCm to 5.7.1, OpenCL applications run fine again.
Another workaround I found is setting the environment variable HSA_ENABLE_SDMA=0. This appears to avoid the problematic code path and OpenCL applications are able to run again.
Bisection
I have tried to bisect the issue and it appears to be related to this new condition introduced between ROCM 5.7.1 and 6.0.0 in src/core/runtime/amd_blit_sdma.cpp:
// Some GFX9 devices require a minimum of 64 DWORDS per ring buffer submission.
if (agent_->isa()->GetVersion() >= core::Isa::Version(9, 0, 0) &&
(agent_->isa()->GetVersion() <= core::Isa::Version(9, 0, 4) ||
agent_->isa()->GetVersion() == core::Isa::Version(9, 0, 12))) {
min_submission_size_ = 256;
}
In particular, this later causes this also newly introduced code to run:
// Pad size is DWORD aligned since all commands are dword aligned.
// Insert NOP header DWORD with value of the number of null DWORDs shifted
// by 16 bits to pad total submission.
if (pad_size) {
memset(command_addr, 0, pad_size);
uint32_t *dword_command_addr = reinterpret_cast<uint32_t*>(command_addr);
dword_command_addr[total_command_size/4] = (pad_size/4 - 1) << 16;
}
Commenting the last statement dword_command_addr[total_command_size/4] = (pad_size/4 - 1) << 16; appears to fix the problem as well. I'm not sure if I'm understanding the code correctly, but I think it is potentially an out-of-bounds write? That would also be consistent with it causing a full GPU reset.
Operating System
Arch Linux updated as of 2024-02-08, Linux kernel 6.7.4.
CPU
AMD Ryzen 7 5700G with Radeon Graphics
GPU
Other
Other
AMD Ryzen 7 5700G with Radeon Graphics - i.e. the integrated APU (gfx90c)
ROCm Version
ROCm 6.0.0
ROCm Component
ROCR-Runtime
Steps to Reproduce
Save the following OpenCL application:
#include <iostream>
#include <vector>
#define CL_HPP_ENABLE_EXCEPTIONS
#define CL_HPP_TARGET_OPENCL_VERSION 200
#include <CL/opencl.hpp>
int main() {
try {
std::vector<cl::Platform> platforms;
cl::Platform::get(&platforms);
if (platforms.empty()) return EXIT_FAILURE;
std::vector<cl::Device> devices;
platforms[0].getDevices(CL_DEVICE_TYPE_GPU, &devices);
if (devices.empty()) return EXIT_FAILURE;
cl::Device device = devices[0];
std::cout << "Vendor: " << device.getInfo<CL_DEVICE_VENDOR>() << std::endl;
std::cout << "Name: " << device.getInfo<CL_DEVICE_NAME>() << std::endl;
cl::Context context(device);
cl::CommandQueue queue(context, device);
std::vector<int> hostData(10, 42);
cl::Buffer buffer(context, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR,
sizeof(int) * hostData.size(), hostData.data());
queue.enqueueWriteBuffer(buffer, CL_TRUE, 0, sizeof(int) * hostData.size(), hostData.data());
std::vector<int> outputData(hostData.size());
queue.enqueueReadBuffer(buffer, CL_TRUE, 0, sizeof(int) * hostData.size(), outputData.data());
std::cout << "Buffer data: ";
for (int i : outputData) {
std::cout << i << " ";
}
std::cout << std::endl;
std::cout << "Success!" << std::endl;
return EXIT_SUCCESS;
} catch (const cl::Error &err) {
std::cerr
<< "OpenCL error: "
<< err.what() << "(" << err.err() << ")"
<< std::endl;
return 1;
}
}
Build it:
$ g++ hello.cpp -o hello -lOpenCL
Run it on a machine with a Ryzen 5700G processor and ROCm 6.0.0:
$ ./hello
Expected Behavior: The program should run correctly and print the following output:
Vendor: Advanced Micro Devices, Inc.
Name: gfx90c:xnack-
Buffer data: 42 42 42 42 42 42 42 42 42 42
Success!
Actual Behavior: The program prints:
Vendor: Advanced Micro Devices, Inc.
Name: gfx90c:xnack-
Then it hangs, and likely causes a GPU reset.
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version: 1.1
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES
==========
HSA Agents
==========
*******
Agent 1
*******
Name: AMD Ryzen 7 5700G with Radeon Graphics
Uuid: CPU-XX
Marketing Name: AMD Ryzen 7 5700G with Radeon Graphics
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 4673
BDFID: 0
Internal Node ID: 0
Compute Unit: 16
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 31717352(0x1e3f7e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 31717352(0x1e3f7e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 31717352(0x1e3f7e8) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:
*******
Agent 2
*******
Name: gfx90c
Uuid: GPU-XX
Marketing Name: AMD Radeon Graphics
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
L2: 1024(0x400) KB
Chip ID: 5688(0x1638)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2000
BDFID: 12288
Internal Node ID: 1
Compute Unit: 8
SIMDs per CU: 4
Shader Engines: 1
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 469
SDMA engine uCode:: 40
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 1048576(0x100000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 1048576(0x100000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx90c:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
Additional Information
dmesg output at the time of the crash:
[ 308.334535] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, signaled seq=1255, emitted seq=1257
[ 308.334876] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process pid 0 thread pid 0
[ 308.335146] amdgpu 0000:30:00.0: amdgpu: GPU reset begin!
[ 308.335154] amdgpu: Failed to suspend process 0x8010
[ 308.545396] amdgpu 0000:30:00.0: amdgpu: MODE2 reset
[ 308.545598] amdgpu 0000:30:00.0: amdgpu: GPU reset succeeded, trying to resume
[ 308.545743] [drm] PCIE GART of 1024M enabled.
[ 308.545746] [drm] PTB located at 0x000000F43FC00000
[ 308.545803] [drm] VRAM is lost due to GPU reset!
[ 308.545804] [drm] PSP is resuming...
[ 309.248396] [drm] reserve 0x400000 from 0xf43f800000 for PSP TMR
[ 309.529565] amdgpu 0000:30:00.0: amdgpu: RAS: optional ras ta ucode is not available
[ 309.540284] amdgpu 0000:30:00.0: amdgpu: RAP: optional rap ta ucode is not available
[ 309.540286] amdgpu 0000:30:00.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[ 309.540288] amdgpu 0000:30:00.0: amdgpu: SMU is resuming...
[ 309.540510] amdgpu 0000:30:00.0: amdgpu: SMU is resumed successfully!
[ 309.541037] [drm] DMUB hardware initialized: version=0x01010028
[ 309.805376] [drm] kiq ring mec 2 pipe 1 q 0
[ 309.807861] [drm] VCN decode and encode initialized successfully(under DPG Mode).
[ 309.807903] [drm] JPEG decode initialized successfully.
[ 309.807906] amdgpu 0000:30:00.0: amdgpu: ring gfx uses VM inv eng 0 on hub 0
[ 309.807908] amdgpu 0000:30:00.0: amdgpu: ring gfx_low uses VM inv eng 1 on hub 0
[ 309.807910] amdgpu 0000:30:00.0: amdgpu: ring gfx_high uses VM inv eng 4 on hub 0
[ 309.807911] amdgpu 0000:30:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 5 on hub 0
[ 309.807912] amdgpu 0000:30:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 6 on hub 0
[ 309.807913] amdgpu 0000:30:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 7 on hub 0
[ 309.807914] amdgpu 0000:30:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 8 on hub 0
[ 309.807915] amdgpu 0000:30:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 9 on hub 0
[ 309.807917] amdgpu 0000:30:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 10 on hub 0
[ 309.807918] amdgpu 0000:30:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 11 on hub 0
[ 309.807919] amdgpu 0000:30:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 12 on hub 0
[ 309.807920] amdgpu 0000:30:00.0: amdgpu: ring kiq_0.2.1.0 uses VM inv eng 13 on hub 0
[ 309.807921] amdgpu 0000:30:00.0: amdgpu: ring sdma0 uses VM inv eng 0 on hub 8
[ 309.807923] amdgpu 0000:30:00.0: amdgpu: ring vcn_dec uses VM inv eng 1 on hub 8
[ 309.807924] amdgpu 0000:30:00.0: amdgpu: ring vcn_enc0 uses VM inv eng 4 on hub 8
[ 309.807925] amdgpu 0000:30:00.0: amdgpu: ring vcn_enc1 uses VM inv eng 5 on hub 8
[ 309.807926] amdgpu 0000:30:00.0: amdgpu: ring jpeg_dec uses VM inv eng 6 on hub 8
[ 309.809563] amdgpu 0000:30:00.0: amdgpu: recover vram bo from shadow start
[ 309.809565] amdgpu 0000:30:00.0: amdgpu: recover vram bo from shadow done
[ 309.809573] amdgpu 0000:30:00.0: amdgpu: GPU reset(1) succeeded!
[ 309.809593] [drm] Skip scheduling IBs!
[ 309.809601] [drm] Skip scheduling IBs!
[ 309.809605] [drm] Skip scheduling IBs!
[ 309.809607] [drm] Skip scheduling IBs!
[ 309.809609] [drm] Skip scheduling IBs!
[ 309.809613] [drm] Skip scheduling IBs!
[ 309.809622] [drm] Skip scheduling IBs!
[ 309.809635] [drm] Skip scheduling IBs!
[ 309.809643] [drm] Skip scheduling IBs!
[ 309.809646] [drm] Skip scheduling IBs!
[ 309.809649] [drm] Skip scheduling IBs!
[ 309.809652] [drm] Skip scheduling IBs!
[ 309.809655] [drm] Skip scheduling IBs!
[ 309.809657] [drm] Skip scheduling IBs!
[ 309.809660] [drm] Skip scheduling IBs!
[ 309.809662] [drm] Skip scheduling IBs!
[ 309.809665] [drm] Skip scheduling IBs!
[ 309.809668] [drm] Skip scheduling IBs!
[ 309.809670] [drm] Skip scheduling IBs!
[ 309.809673] [drm] Skip scheduling IBs!
[ 309.809675] [drm] Skip scheduling IBs!
[ 309.809678] [drm] Skip scheduling IBs!
clinfo output:
Number of platforms 1
Platform Name AMD Accelerated Parallel Processing
Platform Vendor Advanced Micro Devices, Inc.
Platform Version OpenCL 2.1 AMD-APP.dbg (3602.0)
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd cl_amd_event_callback
Platform Extensions function suffix AMD
Platform Host timer resolution 1ns
Platform Name AMD Accelerated Parallel Processing
Number of devices 1
Device Name gfx90c:xnack-
Device Vendor Advanced Micro Devices, Inc.
Device Vendor ID 0x1002
Device Version OpenCL 2.0
Driver Version 3602.0 (HSA1.1,LC)
Device OpenCL C Version OpenCL C 2.0
Device Type GPU
Device Board Name (AMD) AMD Radeon Graphics
Device PCI-e ID (AMD) 0x1638
Device Topology (AMD) PCI-E, 0000:30:00.0
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 8
SIMD per compute unit (AMD) 4
SIMD width (AMD) 16
SIMD instruction width (AMD) 1
Max clock frequency 2000MHz
Graphics IP (AMD) 9.0
Device Partition (core)
Max number of sub-devices 8
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 1024x1024x1024
Max work group size 256
Preferred work group size (AMD) 256
Max work group size (AMD) 1024
Preferred work group size multiple (kernel) 64
Wavefront width (AMD) 64
Preferred / native vector sizes
char 4 / 4
short 2 / 2
int 1 / 1
long 1 / 1
half 1 / 1 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (cl_khr_fp16)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 1073741824 (1024MiB)
Global free memory (AMD) 995328 (972MiB) 995328 (972MiB)
Global memory channels (AMD) 4
Global memory banks per channel (AMD) 4
Global memory bank width (AMD) 256 bytes
Error Correction support No
Max memory allocation 912680544 (870.4MiB)
Unified memory for Host and Device No
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing Yes
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Preferred alignment for atomics
SVM 0 bytes
Global 0 bytes
Local 0 bytes
Max size for global variable 912680544 (870.4MiB)
Preferred total size of global vars 1073741824 (1024MiB)
Global Memory cache type Read/Write
Global Memory cache size 16384 (16KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 134217728 pixels
Max 1D or 2D image array size 8192 images
Base address alignment for 2D image buffers 256 bytes
Pitch alignment for 2D image buffers 256 pixels
Max 2D image size 16384x16384 pixels
Max 3D image size 16384x16384x8192 pixels
Max number of read image args 128
Max number of write image args 8
Max number of read/write image args 64
Max number of pipe args 16
Max active pipe reservations 16
Max pipe packet size 912680544 (870.4MiB)
Local memory type Local
Local memory size 65536 (64KiB)
Local memory size per CU (AMD) 65536 (64KiB)
Local memory banks (AMD) 32
Max number of constant args 8
Max constant buffer size 912680544 (870.4MiB)
Preferred constant buffer size (AMD) 16384 (16KiB)
Max size of kernel argument 1024
Queue properties (on host)
Out-of-order execution No
Profiling Yes
Queue properties (on device)
Out-of-order execution Yes
Profiling Yes
Preferred size 262144 (256KiB)
Max size 8388608 (8MiB)
Max queues on device 1
Max events on device 1024
Prefer user sync for interop Yes
Number of P2P devices (AMD) 0
Profiling timer resolution 1ns
Profiling timer offset since Epoch (AMD) 0ns (Thu Jan 1 00:00:00 1970)
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Thread trace supported (AMD) No
Number of async queues (AMD) 8
Max real-time compute queues (AMD) 8
Max real-time compute units (AMD) 8
printf() buffer size 4194304 (4MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [AMD]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx90c:xnack-
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx90c:xnack-
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx90c:xnack-
NOTE: your OpenCL library only supports OpenCL 2.0,
but some installed platforms support OpenCL 2.1.
Programs using 2.1 features may crash
or behave unexpectedly
Hi @joanbm , Could you please run your test with the follow patch? This fixes the pad_size address -
+++ b/opensrc/hsa-runtime/core/runtime/amd_blit_sdma.cpp
@@ -471,0 +472 @@ hsa_status_t BlitSdma<RingIndexTy, HwIndexMonotonic, SizeToCountOffset, useGCR>:
+ command_addr += trap_command_size_;
@@ -482 +483 @@ hsa_status_t BlitSdma<RingIndexTy, HwIndexMonotonic, SizeToCountOffset, useGCR>:
- dword_command_addr[total_command_size/4] = (pad_size/4 - 1) << 16;
+ dword_command_addr[0] = (pad_size/4 - 1) << 16;
@shwetagkhatri Everything seems to work fine with your patch applied (I tried the test case above, hashcat, and some other simple OpenCL programs), thanks.
Thanks for confirming @joanbm. This fix will be available with ROCM 6.1 release.