pytorch
Incorrect cache-info output with nosmt linux kernel command line
Hello,
I'm using the main branch of today (https://github.com/pytorch/cpuinfo/commit/3c8b1533ac03dd6531ab6e7b9245d488f13a82a5) and I've got incorrect output in cache-info on a machine booted with the "nosmt" kernel command line parameter:
WITH the "nosmt" parameter (note the weird "shared by XX processors" at the end of each line):
[NOSMT]$ ./cache-info
Max cache size (upper bound): 16777216 bytes
L1 instruction cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 97 processors
L1 data cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 97 processors
L2 unified cache: 96 x 512 KB (inclusive), 8-way set associative (1024 sets), 64 byte lines, shared by 97 processors
L3 unified cache: 24 x 16 MB (exclusive), 16-way set associative (16384 sets), 64 byte lines, shared by 100 processors
WITHOUT the "nosmt" parameter:
[SMT]$ ./cache-info
Max cache size (upper bound): 16777216 bytes
L1 instruction cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 2 processors
L1 data cache: 96 x 32 KB, 8-way set associative (64 sets), 64 byte lines, shared by 2 processors
L2 unified cache: 96 x 512 KB (inclusive), 8-way set associative (1024 sets), 64 byte lines, shared by 2 processors
L3 unified cache: 24 x 16 MB (exclusive), 16-way set associative (16384 sets), 64 byte lines, shared by 8 processors
lscpu output WITH the "nosmt" parameter:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-95
Off-line CPU(s) list: 96-191
Thread(s) per core: 1
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
BIOS Vendor ID: Advanced Micro Devices, Inc.
CPU family: 23
Model: 49
Model name: AMD EPYC 7552 48-Core Processor
BIOS Model name: AMD EPYC 7552 48-Core Processor
Stepping: 0
CPU MHz: 2851.220
CPU max MHz: 2200.0000
CPU min MHz: 1500.0000
BogoMIPS: 4400.33
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 16384K
NUMA node0 CPU(s): 0-47
NUMA node1 CPU(s): 48-95
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
lscpu output WITHOUT the "nosmt" parameter:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Thread(s) per core: 2
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
BIOS Vendor ID: Advanced Micro Devices, Inc.
CPU family: 23
Model: 49
Model name: AMD EPYC 7552 48-Core Processor
BIOS Model name: AMD EPYC 7552 48-Core Processor
Stepping: 0
CPU MHz: 2200.000
CPU max MHz: 2200.0000
CPU min MHz: 1500.0000
BogoMIPS: 4400.15
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 16384K
NUMA node0 CPU(s): 0-47,96-143
NUMA node1 CPU(s): 48-95,144-191
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
The difference is are on the On-line/Off-line CPU(s) lists.
It caught me by surprise with a SIGFPE when using the NNPACK project (https://github.com/Maratyszcza/NNPACK/issues/218).
Thank you for your help.
Please find attached the output of cache-info with and without the nosmt option, when cpuinfo is built with debug log level.
cache-info.nosmt.txt cache-info.smt.txt
Looking at the x86 logic (I'm horribly unfamiliar with it, so bear with me here :)), it looks like we'd only accidentally generate N caches if we assigned them each different L1I IDs. Digging through the code paths here, I don't see anywhere we would accidentally mark the same ID as different IDs, so that leads me to believe that our l1i_id calculation fails somehow.
# How we calculate the apic_bits for each cache in `cpuinfo_x86_decode_cache_properties`.
const uint32_t cores = 1 + ((regs.eax >> 14) & UINT32_C(0x00000FFF));
const uint32_t apic_bits = bit_length(cores);
...
# How we later mask to acquire the L1I ID.
const uint32_t apic_id = linux_processors[i].apic_id;
const uint32_t l1i_id = apic_id & ~bit_mask(x86_processor.cache.l1i.apic_bits);
I'd hazard a guess that this means either:
- The
apic_idis wrong for all processors with 'nosmt' enabled? - The
apic_bitsis wrong for all caches with 'nosmt' enabled?
