flatcar
[RFE] AMD CPU scaling drivers
Current situation
Currently AMD's scaling drivers are not compiler in flatcar's kernel (stable & beta channel)
~ # zgrep PSTATE /proc/config.gz
CONFIG_X86_INTEL_PSTATE=y
# CONFIG_X86_AMD_PSTATE is not set
# CONFIG_X86_AMD_PSTATE_UT is not set
Impact
AMD new EPYC + ryzen CPU use acpi-cpufreq instead of amd-pstate epp mode reducing performance.
Ideal future situation
Enabling CONFIG_X86_AMD_PSTATE + CONFIG_HZ_{100,250,300} in kernel config for future release.
Implementation options
Enabling CONFIG_X86_AMD_PSTATE + CONFIG_HZ_{100,250,300} in kernel config for future release.
amd-pstate will be added, thanks for catching this.
Any reason you think CONFIG_HZ=1000 is bad? We also have CONFIG_NO_HZ_IDLE.
Great !
Could we also get X86_AMD_PSTATE_DEFAULT_MODE=3 ?
Any reason you think CONFIG_HZ=1000 is bad? We also have CONFIG_NO_HZ_IDLE.
We would like to avoid a lot of interruption on heavily loaded servers and have the ability to always get 100Hz and periodic_hz. At least to benchmark it
I tested it on one of our server with AMD EPYC 9654 96-Core Processor and 512GB RAM.
amd-pstate-epp working perfectly, better frequencies handling and works out of the box.
Following this flatcar upgrade and all associated changes we got a 2% perf improvements (sysbench results) and especially a drastic decrease in irq/context switches that causes us some issues on real workload.
CONFIG_HZ=100 is maybe a little too low as seen on stress-ng graphs, maybe 250 or 300 is enough + CONFIG_NO_HZ_IDLE to avoid some useless tick.
Flatcar stable 3815.2.1
~ # uname -a
Linux 6.1.81-flatcar #1 SMP PREEMPT_DYNAMIC Sun Mar 17 21:32:50 -00 2024 x86_64 AMD EPYC 9654 96-Core Processor AuthenticAMD GNU/Linux
~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3815.2.1
VERSION_ID=3815.2.1
BUILD_ID=2024-03-17-2158
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3815.2.1 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3815.2.1:*:*:*:*:*:*:*"
~ # zgrep _HZ /proc/config.gz
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
~ # zgrep PSTATE /proc/config.gz
CONFIG_X86_INTEL_PSTATE=y
# CONFIG_X86_AMD_PSTATE is not set
# CONFIG_X86_AMD_PSTATE_UT is not set
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
acpi-cpufreq
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
schedutil
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
2400000 1900000 1500000
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance schedutil
sysbench
Used this sysbench cpu --histogram=on --threads=256 --time=600 run to generate workload easily.
stress-ng
Used this stress-ng -M --all 10 --class=cpu --cpu=256 --perf -t 600 to generate workload.
Flatcar 9999.9.9+amd-pstate-hz100
NB: to use amd-pstate instead of acpi-cpufreq+schedutil I had to modify /usr/share/oem/grub.cfg
~ # cat /usr/share/oem/grub.cfg
# Flatcar GRUB settings
set oem_id="openstack"
set linux_append="$linux_append amd_pstate=active amd_pstate.shared_mem=1"
~ # uname -a
Linux 6.6.21-flatcar #1 SMP PREEMPT_DYNAMIC Tue Mar 19 17:04:32 -00 2024 x86_64 AMD EPYC 9654 96-Core Processor AuthenticAMD GNU/Linux
~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=9999.9.9+amd-pstate-hz100
VERSION_ID=9999.9.9
BUILD_ID=amd-pstate-hz100
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 9999.9.9+amd-pstate-hz100 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:9999.9.9+amd-pstate-hz100:*:*:*:*:*:*:*"
~ # zgrep _HZ /proc/config.gz
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
~ # zgrep PSTATE /proc/config.gz
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_AMD_PSTATE=y
CONFIG_X86_AMD_PSTATE_DEFAULT_MODE=3
# CONFIG_X86_AMD_PSTATE_UT is not set
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
amd-pstate-epp
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/amd_pstate_max_freq
3709000
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave
sysbench
Used this sysbench cpu --histogram=on --threads=256 --time=600 run to generate workload easily.
stress-ng
Used this stress-ng -M --all 10 --class=cpu --cpu=256 --perf -t 600 to generate workload.