procfs
procfs copied to clipboard
eminence
On s390x, test_kernel_stat and test_meminfo fail
I originally saw this while working on an update for the rust-procfs package in Fedora; the package build still happens in a mock chroot, but on real s390x hardware rather than under qemu-user-static emulation. I’ve tried to reproduce it in emulation, below, since I suspect that may be more useful to you.
Working in a Fedora 41 mock chroot with git, cargo, and rust installed, and with network access enabled:
On x86_64, skipping:
process::tests::test_proc_fd_count_runsinglethreadbecause of https://github.com/eminence/procfs/issues/322sys::kernel::random::tests::test_write_wakeup_threshold,process::tests::test_proc_auxv, andprocess::tests::test_proc_status_for_kthreaddbecause they fail in a mock chroot for various reasons
$ uname -srvmo
Linux 6.11.7-300.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 8 19:23:10 UTC 2024 x86_64 GNU/Linux
$ git clone https://github.com/eminence/procfs.git
$ cd procfs
$ git checkout v0.17.0
$ RUST_BACKTRACE=1 cargo test -- --skip-exact \
--skip=process::tests::test_proc_fd_count_runsinglethread \
--skip sys::kernel::random::tests::test_write_wakeup_threshold \
--skip process::tests::test_proc_auxv \
--skip=process::tests::test_proc_status_for_kthreadd
(all tests pass or are skipped/ignored)
This is our “control” build.
Now, repeating in an aarch64 chroot, emulated via qemu-user-static:
$ RUST_BACKTRACE=1 cargo test -- --skip-exact […]
[…]
failures:
---- process::tests::test_proc_exe stdout ----
thread 'process::tests::test_proc_exe' panicked at procfs/src/process/tests.rs:275:5:
assertion `left == right` failed
left: "/usr/bin/qemu-aarch64-static"
right: "/procfs/target/debug/deps/procfs-daeabd3c8934810d"
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:367:5
4: procfs::process::tests::test_proc_exe
at ./src/process/tests.rs:275:5
5: procfs::process::tests::test_proc_exe::{{closure}}
at ./src/process/tests.rs:271:19
6: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
---- tests::test_cpuinfo stdout ----
None
thread 'tests::test_cpuinfo' panicked at procfs/src/lib.rs:684:33:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:148:5
3: core::option::unwrap_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/option.rs:2012:5
4: core::option::Option<T>::unwrap
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/option.rs:972:21
5: procfs::tests::test_cpuinfo
at ./src/lib.rs:684:13
6: procfs::tests::test_cpuinfo::{{closure}}
at ./src/lib.rs:679:22
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
8: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
---- process::tests::test_procinfo stdout ----
thread 'process::tests::test_procinfo' panicked at procfs/src/process/tests.rs:506:52:
called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidInput, error: "unable to parse input: Position(Space, [10])" }
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::result::unwrap_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/result.rs:1700:5
3: core::result::Result<T,E>::unwrap
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/result.rs:1104:23
4: procfs::process::tests::test_procinfo
at ./src/process/tests.rs:506:25
5: procfs::process::tests::test_procinfo::{{closure}}
at ./src/process/tests.rs:494:19
6: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
failures:
process::tests::test_proc_exe
process::tests::test_procinfo
tests::test_cpuinfo
test result: FAILED. 91 passed; 3 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1.27s
Okay, the tests::test_cpuinfo failure is already reported as https://github.com/eminence/procfs/issues/323, the process::tests::test_proc_exe failure seems to be specific to running in emulation, and I don’t know what to say about process::tests::test_procinfo. I didn’t see it in a “real” Fedora build, and none of these are what I am trying to report here.
Repeating the above on ppc64le gives the same result, except that tests::test_cpuinfo doesn’t fail – although it did fail on real ppc64le hardware.
Now, for the subject of this report, on s390x:
[…]
---- tests::test_kernel_stat stdout ----
KernelStats {
total: CpuTime {
user: 83030541,
nice: 1456170471,
system: 13524974,
idle: 3154050570,
iowait: Some(
1543050,
),
irq: Some(
9041546,
),
softirq: Some(
866113,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
cpu_time: [
CpuTime {
user: 5197507,
nice: 90828479,
system: 836449,
idle: 197307712,
iowait: Some(
74260,
),
irq: Some(
558475,
),
softirq: Some(
82892,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4596389,
nice: 91197844,
system: 674628,
idle: 197775509,
iowait: Some(
62124,
),
irq: Some(
548304,
),
softirq: Some(
44540,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3530071,
nice: 90147038,
system: 742351,
idle: 199879912,
iowait: Some(
41213,
),
irq: Some(
526190,
),
softirq: Some(
42161,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 8423397,
nice: 91541602,
system: 1349167,
idle: 192570622,
iowait: Some(
268915,
),
irq: Some(
620041,
),
softirq: Some(
54695,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4052862,
nice: 90933503,
system: 615937,
idle: 198685170,
iowait: Some(
50074,
),
irq: Some(
536112,
),
softirq: Some(
38925,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3940753,
nice: 90860186,
system: 586909,
idle: 198930525,
iowait: Some(
26126,
),
irq: Some(
537879,
),
softirq: Some(
36786,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4180428,
nice: 91013747,
system: 615564,
idle: 198459494,
iowait: Some(
45197,
),
irq: Some(
556439,
),
softirq: Some(
38756,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 8331391,
nice: 91488962,
system: 1236338,
idle: 192896853,
iowait: Some(
224505,
),
irq: Some(
612317,
),
softirq: Some(
53209,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5169403,
nice: 90916745,
system: 843821,
idle: 197260698,
iowait: Some(
75680,
),
irq: Some(
564525,
),
softirq: Some(
49986,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 10122500,
nice: 91362720,
system: 1804961,
idle: 190432249,
iowait: Some(
359176,
),
irq: Some(
652213,
),
softirq: Some(
95041,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4263470,
nice: 91060333,
system: 718679,
idle: 198125533,
iowait: Some(
74775,
),
irq: Some(
565820,
),
softirq: Some(
87917,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5419741,
nice: 91275798,
system: 862684,
idle: 196556631,
iowait: Some(
95307,
),
irq: Some(
590100,
),
softirq: Some(
67471,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3962060,
nice: 90859637,
system: 655854,
idle: 198777343,
iowait: Some(
44810,
),
irq: Some(
547991,
),
softirq: Some(
62077,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3457406,
nice: 90381010,
system: 744833,
idle: 199720763,
iowait: Some(
38593,
),
irq: Some(
530356,
),
softirq: Some(
37793,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 4528047,
nice: 91232485,
system: 645171,
idle: 197878344,
iowait: Some(
32209,
),
irq: Some(
554304,
),
softirq: Some(
37576,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 3855107,
nice: 91070372,
system: 591621,
idle: 198793205,
iowait: Some(
30079,
),
irq: Some(
540472,
),
softirq: Some(
36280,
),
steal: Some(
0,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
],
ctxt: 3912665812,
btime: 1731761843,
processes: 7953721,
procs_running: Some(
4,
),
procs_blocked: Some(
0,
),
}
thread 'tests::test_kernel_stat' panicked at procfs/src/lib.rs:577:9:
assertion `left == right` failed
left: 1
right: 16
stack backtrace:
0: rust_begin_unwind
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:74:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/panicking.rs:367:5
4: procfs::tests::test_kernel_stat
at ./src/lib.rs:577:9
5: procfs::tests::test_kernel_stat::{{closure}}
at ./src/lib.rs:566:26
6: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
7: core::ops::function::FnOnce::call_once
at /builddir/build/BUILD/rust-1.83.0-build/rustc-1.83.0-src/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[…]
failures:
process::tests::test_proc_exe
process::tests::test_procinfo
tests::test_cpuinfo
tests::test_kernel_stat
test result: FAILED. 90 passed; 4 failed; 0 ignored; 0 measured; 4 filtered out; finished in 1.18s
The failures in process::tests::test_proc_exe, process::tests::test_procinfo, and tests::test_cpuinfo are consistent with those on aarch64 and pp664le; tests::test_kernel_stat is one of the failures I was trying to report in this issue. The output on real hardware is similar:
---- tests::test_cpuinfo stdout ----
None
thread 'tests::test_cpuinfo' panicked at src/lib.rs:683:34:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- tests::test_kernel_stat stdout ----
KernelStats {
total: CpuTime {
user: 16790930,
nice: 3522,
system: 3384651,
idle: 233015124,
iowait: Some(
326403,
),
irq: Some(
107465,
),
softirq: Some(
185327,
),
steal: Some(
173507,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
cpu_time: [
CpuTime {
user: 5522045,
nice: 917,
system: 1138502,
idle: 77678704,
iowait: Some(
103183,
),
irq: Some(
53849,
),
softirq: Some(
111186,
),
steal: Some(
58703,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5627933,
nice: 1041,
system: 1124884,
idle: 77670809,
iowait: Some(
113340,
),
irq: Some(
27253,
),
softirq: Some(
37545,
),
steal: Some(
57362,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
CpuTime {
user: 5640952,
nice: 1563,
system: 1121263,
idle: 77665610,
iowait: Some(
109879,
),
irq: Some(
26363,
),
softirq: Some(
36595,
),
steal: Some(
57441,
),
guest: Some(
0,
),
guest_nice: Some(
0,
),
tps: 100,
},
],
ctxt: 555939405,
btime: 1733854463,
processes: 13904974,
procs_running: Some(
3,
),
procs_blocked: Some(
0,
),
}
thread 'tests::test_kernel_stat' panicked at src/lib.rs:577:9:
assertion `left == right` failed
left: 1
right: 3
Unfortunately, I wasn’t able to reproduce the test_meminfo failure in emulation, but it looks like this on real hardware:
---- tests::test_meminfo stdout ----
Meminfo {
mem_total: 26335961088,
mem_free: 4957818880,
mem_available: Some(
25220423680,
),
buffers: 1196032,
cached: 19969449984,
swap_cached: 16384,
active: 5222580224,
inactive: 14961217536,
active_anon: Some(
169172992,
),
inactive_anon: Some(
44261376,
),
active_file: Some(
5053407232,
),
inactive_file: Some(
14916956160,
),
unevictable: Some(
0,
),
mlocked: Some(
0,
),
high_total: None,
high_free: None,
low_total: None,
low_free: None,
mmap_copy: None,
swap_total: 8589930496,
swap_free: 8589144064,
dirty: 666693632,
writeback: 0,
anon_pages: Some(
213184512,
),
mapped: 278470656,
shmem: Some(
278528,
),
slab: 830554112,
s_reclaimable: Some(
554332160,
),
s_unreclaim: Some(
276221952,
),
kernel_stack: Some(
15982592,
),
page_tables: Some(
44687360,
),
secondary_page_tables: Some(
0,
),
quicklists: None,
nfs_unstable: Some(
0,
),
bounce: Some(
0,
),
writeback_tmp: Some(
0,
),
commit_limit: Some(
21757911040,
),
committed_as: 579280896,
vmalloc_total: 547608330240,
vmalloc_used: 64446464,
vmalloc_chunk: 0,
hardware_corrupted: None,
anon_hugepages: None,
shmem_hugepages: None,
shmem_pmd_mapped: None,
cma_total: Some(
0,
),
cma_free: Some(
0,
),
hugepages_total: Some(
0,
),
hugepages_free: Some(
0,
),
hugepages_rsvd: Some(
0,
),
hugepages_surp: Some(
0,
),
hugepagesize: Some(
1048576,
),
direct_map_4k: Some(
7832862720,
),
direct_map_4M: None,
direct_map_2M: None,
direct_map_1G: None,
hugetlb: Some(
0,
),
per_cpu: Some(
20029440,
),
k_reclaimable: Some(
554332160,
),
file_pmd_mapped: None,
file_huge_pages: None,
z_swap: Some(
0,
),
z_swapped: Some(
0,
),
}
thread 'tests::test_meminfo' panicked at src/lib.rs:848:13:
assertion failed: meminfo.hardware_corrupted.is_some()
… and I don’t know what to say about
process::tests::test_procinfo. I didn’t see it in a “real” Fedora build, and none of these are what I am trying to report here.
Actually, this is not quite right. We don’t have a rust-procinfo package in Fedora, so we patch out this test. It’s possible that it would fail in a “real” build after all if we had the necessary dependency.
I don't have access to an s390x environment (though if you have any suggestions about how to easily emulate it, let me know), so I'll need some help here.
For the test_cpuinfo failure, seems like this assertion failed:
assert_eq!(cpuinfo.num_cores(), stat.cpu_time.len());
So can you please share your /proc/cpuinfo file and your /proc/stat file?
For the test_meminfo failure, it seems like this assertion failed:
assert!(meminfo.hardware_corrupted.is_some());
So can you please let me know if /proc/kpagecgroup exists and if you have CONFIG_MEMORY_FAILURE set in your kernel?
The following are from a “scratch” build in Fedora infrastructure on real s390x hardware:
/proc/cpuinfo
vendor_id : IBM/S390
# processors : 2
bogomips per cpu: 3331.00
max thread id : 0
features : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt vxp2 nnpa sie
facilities : 0 1 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 47 48 49 50 51 52 53 54 57 58 59 60 61 64 65 69 71 72 73 74 75 76 77 78 80 81 82 129 130 131 133 134 135 138 139 140 146 147 148 150 151 152 155 156 165 192 193 194 196 197
cache0 : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1 : level=1 type=Instruction scope=Private size=128K line_size=256 associativity=8
cache2 : level=2 type=Unified scope=Private size=32768K line_size=256 associativity=16
cache3 : level=3 type=Unified scope=Shared size=262144K line_size=256 associativity=128
processor 0: version = FF, identification = 525FA8, machine = 3931
processor 1: version = FF, identification = 525FA8, machine = 3931
cpu number : 0
physical id : 0
core id : 0
book id : 0
drawer id : 0
dedicated : 0
address : 0
siblings : 1
cpu cores : 1
version : FF
identification : 525FA8
machine : 3931
cpu MHz dynamic : 5200
cpu MHz static : 5200
cpu number : 1
physical id : 1
core id : 1
book id : 0
drawer id : 0
dedicated : 0
address : 1
siblings : 1
cpu cores : 1
version : FF
identification : 525FA8
machine : 3931
cpu MHz dynamic : 5200
cpu MHz static : 5200
/proc/stat
cpu 8599144 3329 2041924 194091752 529229 91582 184450 106340 0 0
cpu0 4313747 2173 1032878 96965488 269437 56344 135127 52583 0 0
cpu1 4285397 1156 1009045 97126263 259791 35238 49322 53757 0 0
intr 195557018 0 0 0
ctxt 264408362
btime 1733857828
processes 5727211
procs_running 2
procs_blocked 0
softirq 234950034 0 31759716 5300 52874123 21733160 0 633316 51258070 9404 76676945
/proc/kpagecgroup
Exists,
-r--------. 1 root root 0 Dec 22 16:45 /proc/kpagecgroup
CONFIG_MEMORY_FAILURE
I’m not sure how to verify this at runtime, but it looks like we do have this enabled.
For those working on Fedora, installing mock and qemu-user-static is an easy way to emulate other architectures. I can test a package like:
$ fedpkg mockbuild --root fedora-rawhide-s390x
and I can do arbitrary interactive debugging in the chroot with:
$ mock -r fedora-rawhide-s390x -i git cargo rust vim-enhanced
$ mock -r fedora-rawhide-s390x --shell --enable-network
Using mock directly like this doesn’t require me to have an RPM .spec file or have a Fedora account, so this is a straightforward option if you have a machine or VM running Fedora.
Alternatively, you can use qemu-user-system to run a full s390x OS image under emulation. You can get a Fedora Server ISO for s390x at https://alt.fedoraproject.org/alt/. I think virt-manager also supports qemu-user-system emulation.
Podman also supports this (not sure if Docker does as well) so long as there's a container image available:
$ podman run --rm -it --arch s390x fedora:rawhide
[root@79913ff0e543 /]# cat /proc/cpuinfo
vendor_id : IBM/S390
...
Unfortunately, being an emulated container, I think some of this information may be less-truthful than a real one. I get permission denied for /proc/kpagecgroup, but this is a rootless container, so that's probably expected.
