PF_RING
PF_RING copied to clipboard
ntop
kernel panic with pf_ring 8.8.0
Hi
i am using pf_ring 8.8.0 with suricata
Linux version 3.10.0-1160.62.1.el7.x86_64
got kernel panic randomly as below:
[2685596.677234] [PF_RING] failure creating lo in /proc [net=0000000000000000]
[2685596.677240] ------------[ cut here ]------------
[2685596.677242] WARNING: CPU: 7 PID: 13189 at fs/proc/generic.c:399 proc_register+0x8e/0xb0
[2685596.677243] proc_dir_entry '/proc/eth0' already registered
[2685596.677244] Modules linked in: pf_ring(OE+) xt_REDIRECT nf_nat_redirect xt_owner veth vxlan ip6_udp_tunnel udp_tunnel xt_recent xt_nat xt_statistic ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc overlay(T) iptable_mangle AliSecGuard(OE) udp_diag nfsv3 nfs_acl nfs lockd grace fscache tcp_diag inet_diag sunrpc nfit libnvdimm intel_powerclamp ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel parport_pc parport lrw virtio_balloon gf128mul glue_helper i2c_piix4 ablk_helper cryptd pcspkr ip_tables ext4 mbcache jbd2 virtio_net net_failover virtio_console
[2685596.677269] virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel virtio_ring serio_raw virtio drm_panel_orientation_quirks floppy [last unloaded: pf_ring]
[2685596.677278] CPU: 7 PID: 13189 Comm: insmod Kdump: loaded Tainted: G W OE ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
[2685596.677279] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 2221b89 04/01/2014
[2685596.677280] Call Trace:
[2685596.677282] [
crash> bt PID: 13189 TASK: ffff9aa94df8d280 CPU: 7 COMMAND: "insmod" #0 [ffff9a9eeca13788] machine_kexec at ffffffff9a2662f4 #1 [ffff9a9eeca137e8] __crash_kexec at ffffffff9a322b82 #2 [ffff9a9eeca138b8] crash_kexec at ffffffff9a322c70 #3 [ffff9a9eeca138d0] oops_end at ffffffff9a991798 #4 [ffff9a9eeca138f8] die at ffffffff9a230a7b #5 [ffff9a9eeca13928] do_general_protection at ffffffff9a991092 #6 [ffff9a9eeca13960] general_protection at ffffffff9a990718 [exception RIP: pde_subdir_insert+63] RIP: ffffffff9a4cd79f RSP: ffff9a9eeca13a10 RFLAGS: 00010206 RAX: ffff9aabd0300048 RBX: 61613937ffffffff RCX: 0000000000000000 RDX: 0000000000000002 RSI: ffff9ab3b64c7840 RDI: ffff9aabd0300010 RBP: ffff9a9eeca13a30 R8: 000000000001f380 R9: ffffffff9a5888e0 R10: ffff9ab5c0bdf380 R11: fffff3735806da00 R12: ffff9ab3b64c78c5 R13: ffff9aabd0300048 R14: ffff9ab3b64c7840 R15: ffff9aabd0300000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #7 [ffff9a9eeca13a38] proc_register at ffffffff9a4cd99c #8 [ffff9a9eeca13a60] proc_mkdir_data at ffffffff9a4cdb26 #9 [ffff9a9eeca13a88] proc_mkdir at ffffffff9a4cdb95 #10 [ffff9a9eeca13a98] add_device_to_proc at ffffffffc0874b18 [pf_ring] #11 [ffff9a9eeca13ac8] add_device_to_ring_list at ffffffffc0874cfa [pf_ring] #12 [ffff9a9eeca13bc8] ring_notifier at ffffffffc0875178 [pf_ring] #13 [ffff9a9eeca13c90] __register_netdevice_notifier at ffffffff9a8533c8 #14 [ffff9a9eeca13ce0] register_netdevice_notifier at ffffffff9a853533 #15 [ffff9a9eeca13cf0] init_module at ffffffffc0772287 [pf_ring] #16 [ffff9a9eeca13d38] do_one_initcall at ffffffff9a20210a #17 [ffff9a9eeca13d68] load_module at ffffffff9a31ea7a #18 [ffff9a9eeca13eb8] sys_finit_module at ffffffff9a31f0f6 #19 [ffff9a9eeca13f50] system_call_fastpath at ffffffff9a999f92 RIP: 00007f3953d12e29 RSP: 00007ffc2e831e18 RFLAGS: 00010207 RAX: 0000000000000139 RBX: 0000000001bee2c0 RCX: 00007f3953d84f90 RDX: 0000000000000000 RSI: 000000000041a96e RDI: 0000000000000005 RBP: 000000000041a96e R8: 0000000000000000 R9: 00007ffc2e832028 R10: 0000000000000005 R11: 0000000000000206 R12: 0000000000000000 R13: 0000000001bee280 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: 0000000000000139 CS: 0033 SS: 002b crash>
attach bt -f
crash> bt -f PID: 13189 TASK: ffff9aa94df8d280 CPU: 7 COMMAND: "insmod" #0 [ffff9a9eeca13788] machine_kexec at ffffffff9a2662f4 ffff9a9eeca13790: 00009a9eeca137f0 ffff9a9e40000000 ffff9a9eeca137a0: 000000002b001000 ffff9a9e6b001000 ffff9a9eeca137b0: 000000002b000000 ffff9a9eeca137d0 ffff9a9eeca137c0: 000000006caea0ca ffff9a9eeca13968 ffff9a9eeca137d0: ffff9a9eeca137f0 ffff9a9eeca13968 ffff9a9eeca137e0: ffff9a9eeca138b0 ffffffff9a322b82 #1 [ffff9a9eeca137e8] __crash_kexec at ffffffff9a322b82 ffff9a9eeca137f0: ffff9aabd0300000 ffff9ab3b64c7840 ffff9a9eeca13800: ffff9aabd0300048 ffff9ab3b64c78c5 ffff9a9eeca13810: ffff9a9eeca13a30 61613937ffffffff ffff9a9eeca13820: fffff3735806da00 ffff9ab5c0bdf380 ffff9a9eeca13830: ffffffff9a5888e0 000000000001f380 ffff9a9eeca13840: ffff9aabd0300048 0000000000000000 ffff9a9eeca13850: 0000000000000002 ffff9ab3b64c7840 ffff9a9eeca13860: ffff9aabd0300010 ffffffffffffffff ffff9a9eeca13870: ffffffff9a4cd79f 0000000000000010 ffff9a9eeca13880: 0000000000010206 ffff9a9eeca13a10 ffff9a9eeca13890: 0000000000000018 000000006caea0ca ffff9a9eeca138a0: 0000000000000246 000000000000000b ffff9a9eeca138b0: ffff9a9eeca138c8 ffffffff9a322c70 #2 [ffff9a9eeca138b8] crash_kexec at ffffffff9a322c70 ffff9a9eeca138c0: ffffffffb8296f00 ffff9a9eeca138f0 ffff9a9eeca138d0: ffffffff9a991798 #3 [ffff9a9eeca138d0] oops_end at ffffffff9a991798 ffff9a9eeca138d8: ffff9a9eeca13968 0000000000000246 ffff9a9eeca138e8: ffffffff9ac5f10c ffff9a9eeca13920 ffff9a9eeca138f8: ffffffff9a230a7b #4 [ffff9a9eeca138f8] die at ffffffff9a230a7b ffff9a9eeca13900: ffff9a9eeca13968 ffff9aa94df8d280 ffff9a9eeca13910: 0000000000000000 0000000000000000 ffff9a9eeca13920: ffff9a9eeca13958 ffffffff9a991092 #5 [ffff9a9eeca13928] do_general_protection at ffffffff9a991092 ffff9a9eeca13930: 0000000000000001 ffff9ab3b64c78c5 ffff9a9eeca13940: ffff9aabd0300048 ffff9ab3b64c7840 ffff9a9eeca13950: ffff9aabd0300000 ffff9a9eeca13a30 ffff9a9eeca13960: ffffffff9a990718 #6 [ffff9a9eeca13960] general_protection at ffffffff9a990718 [exception RIP: pde_subdir_insert+63] RIP: ffffffff9a4cd79f RSP: ffff9a9eeca13a10 RFLAGS: 00010206 RAX: ffff9aabd0300048 RBX: 61613937ffffffff RCX: 0000000000000000 RDX: 0000000000000002 RSI: ffff9ab3b64c7840 RDI: ffff9aabd0300010 RBP: ffff9a9eeca13a30 R8: 000000000001f380 R9: ffffffff9a5888e0 R10: ffff9ab5c0bdf380 R11: fffff3735806da00 R12: ffff9ab3b64c78c5 R13: ffff9aabd0300048 R14: ffff9ab3b64c7840 R15: ffff9aabd0300000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 ffff9a9eeca13968: ffff9aabd0300000 ffff9ab3b64c7840 ffff9a9eeca13978: ffff9aabd0300048 ffff9ab3b64c78c5 ffff9a9eeca13988: ffff9a9eeca13a30 61613937ffffffff ffff9a9eeca13998: fffff3735806da00 ffff9ab5c0bdf380 ffff9a9eeca139a8: ffffffff9a5888e0 000000000001f380 ffff9a9eeca139b8: ffff9aabd0300048 0000000000000000 ffff9a9eeca139c8: 0000000000000002 ffff9ab3b64c7840 ffff9a9eeca139d8: ffff9aabd0300010 ffffffffffffffff ffff9a9eeca139e8: ffffffff9a4cd79f 0000000000000010 ffff9a9eeca139f8: 0000000000010206 ffff9a9eeca13a10 ffff9a9eeca13a08: 0000000000000018 0000000000000000 ffff9a9eeca13a18: ffff9ab3b64c7840 ffff9aabd0300010 ffff9a9eeca13a28: ffff9aa94df92d60 ffff9a9eeca13a58 ffff9a9eeca13a38: ffffffff9a4cd99c #7 [ffff9a9eeca13a38] proc_register at ffffffff9a4cd99c ffff9a9eeca13a40: ffff9ab3b64c7840 0000000000000000 ffff9a9eeca13a50: ffff9aabd0300000 ffff9a9eeca13a80 ffff9a9eeca13a60: ffffffff9a4cdb26 #8 [ffff9a9eeca13a60] proc_mkdir_data at ffffffff9a4cdb26 ffff9a9eeca13a68: ffff9aabd0300010 ffff9aa94df92d00 ffff9a9eeca13a78: ffff9aa94df92d08 ffff9a9eeca13a90 ffff9a9eeca13a88: ffffffff9a4cdb95 #9 [ffff9a9eeca13a88] proc_mkdir at ffffffff9a4cdb95 ffff9a9eeca13a90: ffff9a9eeca13ac0 ffffffffc0874b18 #10 [ffff9a9eeca13a98] add_device_to_proc at ffffffffc0874b18 [pf_ring] ffff9a9eeca13aa0: ffff9ab297e58000 ffff9aa94df92d00 ffff9a9eeca13ab0: ffff9ab297e58000 0000000000000001 ffff9a9eeca13ac0: ffff9a9eeca13bc0 ffffffffc0874cfa #11 [ffff9a9eeca13ac8] add_device_to_ring_list at ffffffffc0874cfa [pf_ring] ffff9a9eeca13ad0: ffff9a9eeca13fd8 000000000001acc0 ffff9a9eeca13ae0: ffffffff9ae18480 ffffffff9ae8ea98 ffff9a9eeca13af0: 7fffffffffffffff ffffffff9ae8ea90 ffff9a9eeca13b00: ffff9aa94df8d280 000000000000000c ffff9a9eeca13b10: ffff9a9eeca13b20 ffffffff9a98c3d9 ffff9a9eeca13b20: ffff9a9eeca13bc8 ffffffff9a98a0a1 ffff9a9eeca13b30: 7fffffffffffffff ffff9a9eeca13c88 ffff9a9eeca13b40: 0000000000000282 fffff3735806da00 ffff9a9eeca13b50: ffff9ab5c0bdf380 ffffffff9a5888e0 ffff9a9eeca13b60: ffff9aa94df92de0 000000000000013b ffff9a9eeca13b70: ffff9ab4395a3d80 ffff9ab4433f1c00 ffff9a9eeca13b80: 0000000000000001 ffff9aabcff00028 ffff9a9eeca13b90: 000000006caea0ca ffff9ab297e58000 ffff9a9eeca13ba0: ffffffffc088e830 ffffffffc087df00 ffff9a9eeca13bb0: 0000000000000000 ffffffffc08919e0 ffff9a9eeca13bc0: ffff9a9eeca13c88 ffffffffc0875178 #12 [ffff9a9eeca13bc8] ring_notifier at ffffffffc0875178 [pf_ring] ffff9a9eeca13bd0: ffffffff9a98c7ad 0000000000000001 ffff9a9eeca13be0: ffff9aa94df8d280 ffffffff00000001 ffff9a9eeca13bf0: dead000000000100 dead000000000200 ffff9a9eeca13c00: 000000006caea0ca ffffffff9ae7a380 ffff9a9eeca13c10: ffffffff9ae8ea90 ffffffff9af5e300 ffff9a9eeca13c20: ffffffff9ae8ea60 ffff9a9eeca13c78 ffff9a9eeca13c30: ffffffff9a359e99 ffff9a9eeca13c60 ffff9a9eeca13c40: ffffffff9a426af7 ffffffff9ae8ea88 ffff9a9eeca13c50: ffff9ab466150000 000000006caea0ca ffff9a9eeca13c60: ffff9ab297e58000 ffffffffc087d040 ffff9a9eeca13c70: ffffffffc0874ee0 0000000000000000 ffff9a9eeca13c80: ffff9ab450919550 ffff9a9eeca13cd8 ffff9a9eeca13c90: ffffffff9a8533c8 #13 [ffff9a9eeca13c90] __register_netdevice_notifier at ffffffff9a8533c8 ffff9a9eeca13c98: ffffffff9b464b00 ffff9ab450919480 ffff9a9eeca13ca8: ffff9ab500000000 ffffffff9ae18020 ffff9a9eeca13cb8: ffff9ab16c4627a0 ffffffffc0772000 ffff9a9eeca13cc8: 0000000000000000 ffffffffc087d3c0 ffff9a9eeca13cd8: ffff9a9eeca13ce8 ffffffff9a853533 #14 [ffff9a9eeca13ce0] register_netdevice_notifier at ffffffff9a853533 ffff9a9eeca13ce8: ffff9a9eeca13d30 ffffffffc0772287 #15 [ffff9a9eeca13cf0] init_module at ffffffffc0772287 [pf_ring] ffff9a9eeca13cf8: 000000006c462c01 0000000000000001 ffff9a9eeca13d08: 0000000000000032 ffffffffc0772000 ffff9a9eeca13d18: ffffffff9ae18020 ffff9ab16c4627a0 ffff9a9eeca13d28: ffffffffc0772000 ffff9a9eeca13d60 ffff9a9eeca13d38: ffffffff9a20210a #16 [ffff9a9eeca13d38] do_one_initcall at ffffffff9a20210a ffff9a9eeca13d40: ffffffffc087d3d8 ffff9a9eeca13ec0 ffff9a9eeca13d50: 0000000000000000 0000000000000001 ffff9a9eeca13d60: ffff9a9eeca13eb0 ffffffff9a31ea7a #17 [ffff9a9eeca13d68] load_module at ffffffff9a31ea7a ffff9a9eeca13d70: ffffffff9a5b4cd0 ffff9aabbaeaca10 ffff9a9eeca13d80: ffffbdab4d0cff10 ffff9aa94df8d280 ffff9a9eeca13d90: ffff9aa94df8d280 ffffffffc0773000 ffff9a9eeca13da0: 0000000000002718 ffffbdab4d0cf610 ffff9a9eeca13db0: ffffffffc087d3d8 ffffffffc087d5d0 ffff9a9eeca13dc0: ffff9aac1d983df0 ffff9aac1d983de0 ffff9a9eeca13dd0: ffff9aab00000001 ffff9aaf7d4ec300 ffff9a9eeca13de0: 00000000baeaca00 ffffffff9ad46d08 ffff9a9eeca13df0: ffff9a9eeca13e00 ffffffff9a450b73 ffff9a9eeca13e00: ffff9a9eeca13eb0 ffffffff9a31a603 ffff9a9eeca13e10: ffffffffc0876030 0000000000000002 ffff9a9eeca13e20: 0000000000000000 0000000000000000 ffff9a9eeca13e30: 0000000000000000 0000000000000000 ffff9a9eeca13e40: 00006c656e72656b 0000000000000000 ffff9a9eeca13e50: 0000000000000000 0000000000000000 ffff9a9eeca13e60: 0000000000000000 0000000000000000 ffff9a9eeca13e70: 0000000000000000 000000006caea0ca ffff9a9eeca13e80: 000000006caea0ca ffff9a9eeca13ec0 ffff9a9eeca13e90: 0000000000000000 0000000000000005 ffff9a9eeca13ea0: 000000000041a96e 0000000000000000 ffff9a9eeca13eb0: ffff9a9eeca13f48 ffffffff9a31f0f6 #18 [ffff9a9eeca13eb8] sys_finit_module at ffffffff9a31f0f6 ffff9a9eeca13ec0: ffffbdab4d09f000 0000000000030fd0 ffff9a9eeca13ed0: ffffbdab4d0cf610 ffffbdab4d0cf4a8 ffff9a9eeca13ee0: ffffbdab4d0bbcf8 000000000002d000 ffff9a9eeca13ef0: 000000000002e650 0000000000000000 ffff9a9eeca13f00: 0000000000000000 0000002500000024 ffff9a9eeca13f10: 0000001c0000001f 0000000000000016 ffff9a9eeca13f20: 000000006caea0ca 0000000000000000 ffff9a9eeca13f30: 0000000000000000 0000000000000000 ffff9a9eeca13f40: 0000000000000000 0000000000000000 ffff9a9eeca13f50: ffffffff9a999f92 #19 [ffff9a9eeca13f50] system_call_fastpath at ffffffff9a999f92 RIP: 00007f3953d12e29 RSP: 00007ffc2e831e18 RFLAGS: 00010207 RAX: 0000000000000139 RBX: 0000000001bee2c0 RCX: 00007f3953d84f90 RDX: 0000000000000000 RSI: 000000000041a96e RDI: 0000000000000005 RBP: 000000000041a96e R8: 0000000000000000 R9: 00007ffc2e832028 R10: 0000000000000005 R11: 0000000000000206 R12: 0000000000000000 R13: 0000000001bee280 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: 0000000000000139 CS: 0033 SS: 002b crash>
Is this issue related to containers? If I don't care about the network interfaces of the container, is there any configuration to workaround？
@wz200210 please provide more details about the setup. e.g. are you running in a container? what capture interfaces are you using? are you running any zc driver? model? Thank you.
Here is the suricata config
I run suricata in a virtualmachine , some other containers also run on this virtualmachine
PFring capture eth0 only. default mode.
seconds before the crash something like page alloc failed accur like this
[2685596.666280] insmod: page allocation failure: order:8, mode:0xc0d0
596.567874] [PF_RING] Welcome to PF_RING 8.8.0 ($Revision: unknown$)
(C) 2004-24 ntop
[2685596.567876] LINUX_VERSION_CODE 00030A00
[2685596.567878] [PF_RING] Min # ring slots 4096
[2685596.567879] [PF_RING] Slot version 21
[2685596.567880] [PF_RING] Capture TX Yes [RX+TX]
[2685596.567881] [PF_RING] IP Defragment No
[2685596.567891] NET: Registered protocol family 27
[2685596.666280] insmod: page allocation failure: order:8, mode:0xc0d0
[2685596.666284] CPU: 0 PID: 13189 Comm: insmod Kdump: loaded Tainted: G W OE ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
[2685596.666285] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 2221b89 04/01/2014
[2685596.666286] Call Trace:
[2685596.666292] [
OS: CentOS 7.9 Linux version 3.10.0-1160.62.1.el7.x86_64
NET info while crash:
crash> net NET_DEVICE NAME IP ADDRESS(ES) ffff9ab5c0db4000 lo 127.0.0.1 ffff9ab5bd2f5000 eth0 10.180.0.64 ffff9ab5b6338000 docker0 172.17.0.1 ffff9ab4660c0000 flannel.1 10.248.5.0 ffff9ab439710000 cni0 10.248.5.1 ffff9ab432907000 veth220e9b20 ffff9ab4476da000 veth77830082 ffff9ab46650c000 vethf7fe77d7 ffff9ab40e2f9000 vethca39530f ffff9ab428f07000 veth6afab18b ffff9ab2af94f000 vethdc962263 ffff9ab420417000 veth1f65ed3c ffff9ab3a9f44000 vethe5d9d559 ffff9a9efb9ba000 veth12beb84a ffff9ab0990fc000 vethe464b47d ffff9ab27d182000 veth833cbda2 ffff9aaf1f75d000 veth80a0e945 ffff9aaf8923c000 vethd7d083c6 ffff9aabd3c86000 veth5cb23c62 ffff9aabdb068000 veth343137be ffff9ab04756d000 vethb0bfe5da ffff9aa0b1941000 vethb7c87aaf ffff9a9ec0ba0000 veth8e82cf6e ffff9a9f4ff23000 veth71e3a523 ffff9a9f53a7f000 veth9aeeac1d ffff9a9e9abc3000 veth1499a6e5 ffff9aa454f8c000 veth7f56184a ffff9a9f5c426000 veth73601f19 ffff9aa9986f6000 vetha1fa65c0 ffff9a9e67ef3000 vethd827fa9e ffff9ab59c714000 veth3d620855 ffff9a9eb33de000 vethc53cba3c ffff9ab4661fb000 vetha5381e0b
I've tried to insmod/rmmod pfring, turn on/off eth0 promisc, up and down other containers repeatablly continuously at the same time. But it's hard to reproduce the issue
I Got these information when I analysis the vmcore：
seams like the parent dir of /proc/net/pf_ring/dev/lo 's proc_dir_entry is broken
Crash near pde_subdir_insert
/usr/src/debug/kernel-3.10.0-1160.62.1.el7/linux-3.10.0-1160.62.1.el7.x86_64/fs/proc/generic.c: 397 0xffffffff9a4cd991 <proc_register+65>: mov %r12,%rsi 0xffffffff9a4cd994 <proc_register+68>: mov %r13,%rdi 0xffffffff9a4cd997 <proc_register+71>: callq 0xffffffff9a4cd760 <pde_subdir_insert>
RSI: ffff9ab3b64c7840 RDI: ffff9aabd0300010
Register RSI: ffff9ab3b64c7840 /proc/net/pf_ring/dev/lo： looks fine，and it's parent point to 0xffff9aabd0300010 same as register RDI
crash> struct proc_dir_entry ffff9ab3b64c7840 struct proc_dir_entry { low_ino = 4026534068, mode = 16749, nlink = 2, uid = { val = 0 }, gid = { val = 0 }, size = 0, proc_iops = 0xffffffff9aa35ec0, proc_fops = 0xffffffff9aa35f80, parent = 0xffff9aabd0300010, subdir = { rb_node = 0x0 }, subdir_node = { __rb_parent_color = 0, rb_right = 0x0, rb_left = 0x0 }, data = 0x0, count = { counter = 1 }, in_use = { counter = 0 }, pde_unload_completion = 0x0, pde_openers = { next = 0xffff9ab3b64c78b0, prev = 0xffff9ab3b64c78b0 }, pde_unload_lock = { { rlock = { raw_lock = { val = { counter = 0 } } } } }, namelen = 2 '\002', name = 0xffff9ab3b64c78c5 "lo" }
Register RDI: ffff9aabd0300010 /proc/net/pf_ring/dev: seams the parent node of lo is broken
crash> struct proc_dir_entry ffff9aabd0300010 struct proc_dir_entry { low_ino = 3492806672, mode = 39595, nlink = 3492806673, uid = { val = 4294941355 }, gid = { val = 0 }, size = 0, proc_iops = 0x100000101, proc_fops = 0x0, parent = 0x0, subdir = { rb_node = 0x61613937ffffffff }, subdir_node = { __rb_parent_color = 8724194240, rb_right = 0xffff9ab5b25fa200, rb_left = 0xffff9ab3de52bce0 }, data = 0xffff9aabd03004c0, count = { counter = 0 }, in_use = { counter = 0 }, pde_unload_completion = 0x0, pde_openers = { next = 0xffff9ab5b76732f8, prev = 0xffff9ab5b76732e8 }, pde_unload_lock = { { rlock = { raw_lock = { val = { counter = 0 } } } } }, namelen = 6 '\006', name = 0xffff9aabd0300095 "" }
Thank you for the trace, I am taking a look at the code to see if there is any obvious bug, however please note Centos 7 is no longer supported (we still have the repo online, but no updates are released and we are unable to support it). Thus I suggest to move to a more recent version and install latest pf_ring stable (9.0) or dev (9.1) to be able to get updates and fixes.
It would be great if you have the chance to clone the dev branch from github and test it. In the meantime I also added additional safety checks on that branch to figure out what's going on.
thanks for your great support! this is my fork https://github.com/wz200210/PF_RING, and i have noticed your safety check commit
The test system is provided by cloud vendor so i am not able to update the system to a newer version. it will still be Centos 7 It will take some time for me to reproduce the issue
hello cardigliano
I try to reproduce this issue with your safety check commit with debug mode I run 3 scripts at the same time： script A：insmod/rmmod pfring, repeatablly continuously script B：up and down common containers repeatablly continuously script C：up and down host-net containers repeatablly continuously
add_device_to_ring_list crash is not reproduced but i got another remove_device_from_proc crash on two VM：
[314159.467897] [PF_RING] Welcome to PF_RING 8.8.0 ($Revision: unknown$)
(C) 2004-24 ntop
[314159.467910] LINUX_VERSION_CODE 00030A00
[314159.467917] [PF_RING] Min # ring slots 4096
[314159.467923] [PF_RING] Slot version 21
[314159.467929] [PF_RING] Capture TX Yes [RX+TX]
[314159.467936] [PF_RING] IP Defragment No
[314159.468004] NET: Registered protocol family 27
[314159.616287] [PF_RING][DEBUG] ring_net_init:9084 init network namespace [net=ffffffff88316180]
[314159.616398] [PF_RING][DEBUG] ring_proc_init:1893 registered /proc/net/pf_ring [net=ffffffff88316180]
[314159.616885] [PF_RING][DEBUG] ring_net_init:9084 init network namespace [net=ffff9793b76e3d80]
[314159.616991] [PF_RING][DEBUG] ring_proc_init:1893 registered /proc/net/pf_ring [net=ffff9793b76e3d80]
[314159.617333] [PF_RING][DEBUG] ring_net_init:9084 init network namespace [net=ffff979706f45200]
[314159.617438] [PF_RING][DEBUG] ring_proc_init:1893 registered /proc/net/pf_ring [net=ffff979706f45200]
[314159.617478] [PF_RING] lo: NETDEV_REGISTER Type=772 IfIndex=1 Ptr=ffff979475d18000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617495] [PF_RING][DEBUG] ring_notifier:8966 lo: [REGISTER][ifindex: 1]
[314159.617518] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=lo ifindex=1 (1)
[314159.617530] [PF_RING][DEBUG] add_device_to_proc:8761 created lo in /proc [net=ffffffff88316180] [entry=ffff97968e68fc00]
[314159.617542] [PF_RING][DEBUG] add_device_to_proc:8775 created lo/info in /proc [net=ffffffff88316180] [entry=ffff97968e68ff00]
[314159.617551] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device lo does NOT support hardware packet filtering (2)
[314159.617564] [PF_RING] lo: NETDEV_UP Type=772 IfIndex=1 Ptr=ffff979475d18000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617578] [PF_RING] ens192: NETDEV_REGISTER Type=1 IfIndex=2 Ptr=ffff97972bb44000 Namespace=ffffffff88316180 Addr=0050569E2007
[314159.617586] [PF_RING][DEBUG] ring_notifier:8966 ens192: [REGISTER][ifindex: 2]
[314159.617604] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=ens192 ifindex=2 (1)
[314159.617616] [PF_RING][DEBUG] add_device_to_proc:8761 created ens192 in /proc [net=ffffffff88316180] [entry=ffff97968e68f240]
[314159.617632] [PF_RING][DEBUG] add_device_to_proc:8775 created ens192/info in /proc [net=ffffffff88316180] [entry=ffff97968e68f900]
[314159.617640] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device ens192 does NOT support hardware packet filtering (2)
[314159.617653] [PF_RING] ens192: NETDEV_UP Type=1 IfIndex=2 Ptr=ffff97972bb44000 Namespace=ffffffff88316180 Addr=0050569E2007
[314159.617667] [PF_RING] docker0: NETDEV_REGISTER Type=1 IfIndex=3 Ptr=ffff97972d0b6000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617675] [PF_RING][DEBUG] ring_notifier:8966 docker0: [REGISTER][ifindex: 3]
[314159.617694] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=docker0 ifindex=3 (1)
[314159.617705] [PF_RING][DEBUG] add_device_to_proc:8761 created docker0 in /proc [net=ffffffff88316180] [entry=ffff97968e68f480]
[314159.617718] [PF_RING][DEBUG] add_device_to_proc:8775 created docker0/info in /proc [net=ffffffff88316180] [entry=ffff97940ddc0240]
[314159.617727] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device docker0 does NOT support hardware packet filtering (2)
[314159.617740] [PF_RING] docker0: NETDEV_UP Type=1 IfIndex=3 Ptr=ffff97972d0b6000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617753] [PF_RING] flannel.1: NETDEV_REGISTER Type=1 IfIndex=4 Ptr=ffff97972d344000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617766] [PF_RING][DEBUG] ring_notifier:8966 flannel.1: [REGISTER][ifindex: 4]
[314159.617786] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=flannel.1 ifindex=4 (1)
[314159.617798] [PF_RING][DEBUG] add_device_to_proc:8761 created flannel.1 in /proc [net=ffffffff88316180] [entry=ffff97940ddc0b40]
[314159.617810] [PF_RING][DEBUG] add_device_to_proc:8775 created flannel.1/info in /proc [net=ffffffff88316180] [entry=ffff97940ddc03c0]
[314159.617819] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device flannel.1 does NOT support hardware packet filtering (2)
[314159.617832] [PF_RING] flannel.1: NETDEV_UP Type=1 IfIndex=4 Ptr=ffff97972d344000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617845] [PF_RING] cni0: NETDEV_REGISTER Type=1 IfIndex=5 Ptr=ffff97972d0b2000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617853] [PF_RING][DEBUG] ring_notifier:8966 cni0: [REGISTER][ifindex: 5]
[314159.617872] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=cni0 ifindex=5 (1)
[314159.617883] [PF_RING][DEBUG] add_device_to_proc:8761 created cni0 in /proc [net=ffffffff88316180] [entry=ffff97940ddc0e40]
[314159.617899] [PF_RING][DEBUG] add_device_to_proc:8775 created cni0/info in /proc [net=ffffffff88316180] [entry=ffff97940ddc0180]
[314159.617906] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device cni0 does NOT support hardware packet filtering (2)
[314159.617919] [PF_RING] cni0: NETDEV_UP Type=1 IfIndex=5 Ptr=ffff97972d0b2000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617933] [PF_RING] vethcd103a93: NETDEV_REGISTER Type=1 IfIndex=6 Ptr=ffff97972d234000 Namespace=ffffffff88316180 Addr=000000000000
[314159.617941] [PF_RING][DEBUG] ring_notifier:8966 vethcd103a93: [REGISTER][ifindex: 6]
[314159.617960] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=vethcd103a93 ifindex=6 (1)
[314159.617972] [PF_RING][DEBUG] add_device_to_proc:8761 created vethcd103a93 in /proc [net=ffffffff88316180] [entry=ffff97940ddc0540]
[314159.617984] [PF_RING][DEBUG] add_device_to_proc:8775 created vethcd103a93/info in /proc [net=ffffffff88316180] [entry=ffff97940ddc0300]
[314159.617993] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device vethcd103a93 does NOT support hardware packet filtering (2)
[314159.618006] [PF_RING] vethcd103a93: NETDEV_UP Type=1 IfIndex=6 Ptr=ffff97972d234000 Namespace=ffffffff88316180 Addr=000000000000
[314159.618024] [PF_RING] veth930dc200: NETDEV_REGISTER Type=1 IfIndex=7 Ptr=ffff9796ab392000 Namespace=ffffffff88316180 Addr=000000000000
[314159.618032] [PF_RING][DEBUG] ring_notifier:8966 veth930dc200: [REGISTER][ifindex: 7]
[314159.618055] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=veth930dc200 ifindex=7 (1)
[314159.618069] [PF_RING][DEBUG] add_device_to_proc:8761 created veth930dc200 in /proc [net=ffffffff88316180] [entry=ffff97940ddc0c00]
[314159.618085] [PF_RING][DEBUG] add_device_to_proc:8775 created veth930dc200/info in /proc [net=ffffffff88316180] [entry=ffff97940ddc0900]
[314159.618093] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device veth930dc200 does NOT support hardware packet filtering (2)
[314159.618106] [PF_RING] veth930dc200: NETDEV_UP Type=1 IfIndex=7 Ptr=ffff9796ab392000 Namespace=ffffffff88316180 Addr=000000000000
[314159.618120] [PF_RING] lo: NETDEV_REGISTER Type=772 IfIndex=1 Ptr=ffff979688879000 Namespace=ffff9793b76e3d80 Addr=000000000000
[314159.618131] [PF_RING][DEBUG] ring_notifier:8966 lo: [REGISTER][ifindex: 1]
[314159.618260] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=lo ifindex=1 (1)
[314159.618286] [PF_RING][DEBUG] add_device_to_proc:8761 created lo in /proc [net=ffff9793b76e3d80] [entry=ffff97940ddc06c0]
[314159.618302] [PF_RING][DEBUG] add_device_to_proc:8775 created lo/info in /proc [net=ffff9793b76e3d80] [entry=ffff97940ddc00c0]
[314159.618314] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device lo does NOT support hardware packet filtering (2)
[314159.618328] [PF_RING] lo: NETDEV_UP Type=772 IfIndex=1 Ptr=ffff979688879000 Namespace=ffff9793b76e3d80 Addr=000000000000
[314159.618345] [PF_RING] eth0: NETDEV_REGISTER Type=1 IfIndex=3 Ptr=ffff97972d236000 Namespace=ffff9793b76e3d80 Addr=000000000000
[314159.618357] [PF_RING][DEBUG] ring_notifier:8966 eth0: [REGISTER][ifindex: 3]
[314159.618376] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=eth0 ifindex=3 (1)
[314159.618395] [PF_RING][DEBUG] add_device_to_proc:8761 created eth0 in /proc [net=ffff9793b76e3d80] [entry=ffff97940ddc0000]
[314159.618411] [PF_RING][DEBUG] add_device_to_proc:8775 created eth0/info in /proc [net=ffff9793b76e3d80] [entry=ffff97940ddc0d80]
[314159.618423] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device eth0 does NOT support hardware packet filtering (2)
[314159.618438] [PF_RING] eth0: NETDEV_UP Type=1 IfIndex=3 Ptr=ffff97972d236000 Namespace=ffff9793b76e3d80 Addr=000000000000
[314159.618453] [PF_RING] lo: NETDEV_REGISTER Type=772 IfIndex=1 Ptr=ffff979688b96000 Namespace=ffff979706f45200 Addr=000000000000
[314159.618461] [PF_RING][DEBUG] ring_notifier:8966 lo: [REGISTER][ifindex: 1]
[314159.618484] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=lo ifindex=1 (1)
[314159.618500] [PF_RING][DEBUG] add_device_to_proc:8761 created lo in /proc [net=ffff979706f45200] [entry=ffff97940ddc0480]
[314159.618514] [PF_RING][DEBUG] add_device_to_proc:8775 created lo/info in /proc [net=ffff979706f45200] [entry=ffff979424b1f600]
[314159.618531] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device lo does NOT support hardware packet filtering (2)
[314159.618545] [PF_RING] lo: NETDEV_UP Type=772 IfIndex=1 Ptr=ffff979688b96000 Namespace=ffff979706f45200 Addr=000000000000
[314159.618561] [PF_RING] eth0: NETDEV_REGISTER Type=1 IfIndex=3 Ptr=ffff9796ab393000 Namespace=ffff979706f45200 Addr=000000000000
[314159.618573] [PF_RING][DEBUG] ring_notifier:8966 eth0: [REGISTER][ifindex: 3]
[314159.618592] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=eth0 ifindex=3 (1)
[314159.618606] [PF_RING][DEBUG] add_device_to_proc:8761 created eth0 in /proc [net=ffff979706f45200] [entry=ffff979424b1f540]
[314159.618618] [PF_RING][DEBUG] add_device_to_proc:8775 created eth0/info in /proc [net=ffff979706f45200] [entry=ffff979424b1fa80]
[314159.618630] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device eth0 does NOT support hardware packet filtering (2)
[314159.618644] [PF_RING] eth0: NETDEV_UP Type=1 IfIndex=3 Ptr=ffff9796ab393000 Namespace=ffff979706f45200 Addr=000000000000
[314159.618652] [PF_RING] pf_ring initialized correctly
[314160.254231] [PF_RING] docker0: NETDEV_CHANGE Type=1 IfIndex=3 Ptr=ffff97972d0b6000 Namespace=ffffffff88316180 Addr=000000000000
[314162.178177] [PF_RING] veth361ac4f: NETDEV_POST_INIT Type=1 IfIndex=7744 Ptr=ffff97973f724000 Namespace=ffffffff88316180 Addr=000000000000
[314162.178580] [PF_RING] veth361ac4f: NETDEV_REGISTER Type=1 IfIndex=7744 Ptr=ffff97973f724000 Namespace=ffffffff88316180 Addr=000000000000
[314162.178589] [PF_RING][DEBUG] ring_notifier:8966 veth361ac4f: [REGISTER][ifindex: 7744]
[314162.178612] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=veth361ac4f ifindex=7744 (1)
[314162.178627] [PF_RING][DEBUG] add_device_to_proc:8761 created veth361ac4f in /proc [net=ffffffff88316180] [entry=ffff97972a9eeb40]
[314162.178638] [PF_RING][DEBUG] add_device_to_proc:8775 created veth361ac4f/info in /proc [net=ffffffff88316180] [entry=ffff97972a9eec00]
[314162.178646] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device veth361ac4f does NOT support hardware packet filtering (2)
[314162.178961] [PF_RING] vethed0ba5d: NETDEV_POST_INIT Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.179366] [PF_RING] vethed0ba5d: NETDEV_REGISTER Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.179374] [PF_RING][DEBUG] ring_notifier:8966 vethed0ba5d: [REGISTER][ifindex: 7745]
[314162.179393] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=vethed0ba5d ifindex=7745 (1)
[314162.179406] [PF_RING][DEBUG] add_device_to_proc:8761 created vethed0ba5d in /proc [net=ffffffff88316180] [entry=ffff979406cda0c0]
[314162.179417] [PF_RING][DEBUG] add_device_to_proc:8775 created vethed0ba5d/info in /proc [net=ffffffff88316180] [entry=ffff979406cdab40]
[314162.179425] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device vethed0ba5d does NOT support hardware packet filtering (2)
[314162.180333] docker0: port 1(vethed0ba5d) entered blocking state
[314162.180343] docker0: port 1(vethed0ba5d) entered disabled state
[314162.180364] [PF_RING] vethed0ba5d: NETDEV_JOIN Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.180468] [PF_RING] vethed0ba5d: Unknown msg 26 Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.180499] [PF_RING] vethed0ba5d: Unknown msg 21 Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.180600] device vethed0ba5d entered promiscuous mode
[314162.180706] [PF_RING] docker0: NETDEV_FEAT_CHANGE Type=1 IfIndex=3 Ptr=ffff97972d0b6000 Namespace=ffffffff88316180 Addr=000000000000
[314162.180938] [PF_RING] vethed0ba5d: NETDEV_PRE_UP Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.181275] [PF_RING] vethed0ba5d: NETDEV_UP Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.181289] IPv6: ADDRCONF(NETDEV_UP): vethed0ba5d: link is not ready
[314162.181298] docker0: port 1(vethed0ba5d) entered blocking state
[314162.181307] docker0: port 1(vethed0ba5d) entered forwarding state
[314162.182931] [PF_RING] vethed0ba5d: NETDEV_CHANGE Type=1 IfIndex=7745 Ptr=ffff97973f721000 Namespace=ffffffff88316180 Addr=000000000000
[314162.182945] docker0: port 1(vethed0ba5d) entered disabled state
[314162.183287] [PF_RING] docker0: NETDEV_CHANGE Type=1 IfIndex=3 Ptr=ffff97972d0b6000 Namespace=ffffffff88316180 Addr=000000000000
[314162.353274] IPVS: Creating netns size=2048 id=3940
[314162.353526] [PF_RING][DEBUG] ring_net_init:9084 init network namespace [net=ffff97972d0a9480]
[314162.353622] [PF_RING][DEBUG] ring_proc_init:1893 registered /proc/net/pf_ring [net=ffff97972d0a9480]
[314162.353673] [PF_RING] lo: NETDEV_POST_INIT Type=772 IfIndex=1 Ptr=ffff97973fd2a000 Namespace=ffff97972d0a9480 Addr=000000000000
[314162.354023] [PF_RING] lo: NETDEV_REGISTER Type=772 IfIndex=1 Ptr=ffff97973fd2a000 Namespace=ffff97972d0a9480 Addr=000000000000
[314162.354031] [PF_RING][DEBUG] ring_notifier:8966 lo: [REGISTER][ifindex: 1]
[314162.354052] [PF_RING][DEBUG] add_device_to_ring_list:8799 adding dev=lo ifindex=1 (1)
[314162.354069] [PF_RING][DEBUG] add_device_to_proc:8761 created lo in /proc [net=ffff97972d0a9480] [entry=ffff979335b65c00]
[314162.354108] [PF_RING][DEBUG] add_device_to_proc:8775 created lo/info in /proc [net=ffff97972d0a9480] [entry=ffff979335b65000]
[314162.354115] [PF_RING][DEBUG] add_device_to_ring_list:8835 Device lo does NOT support hardware packet filtering (2)
[314162.372145] [PF_RING] lo: NETDEV_PRE_UP Type=772 IfIndex=1 Ptr=ffff97973fd2a000 Namespace=ffff97972d0a9480 Addr=000000000000
[314162.372249] [PF_RING] lo: NETDEV_UP Type=772 IfIndex=1 Ptr=ffff97973fd2a000 Namespace=ffff97972d0a9480 Addr=000000000000
[314162.745746] [PF_RING][DEBUG] remove_device_from_proc:8664 removing lo/info from /proc [net=ffff97972d0a9480] [entry=ffff979335b65000]
[314162.745764] [PF_RING][DEBUG] remove_device_from_proc:8673 removing lo from /proc [net=ffff97972d0a9480] [entry=ffff979335b65c00]
[314162.745779] [PF_RING] veth361ac4f: NETDEV_UNREGISTER Type=1 IfIndex=7744 Ptr=ffff97973f724000 Namespace=ffffffff88316180 Addr=000000000000
[314162.745787] [PF_RING][DEBUG] remove_device_from_proc:8664 removing vethed0ba5d/info from /proc [net=ffffffff88316180] [entry=ffff979406cdab40]
[314162.745803] [PF_RING][DEBUG] remove_device_from_proc:8673 removing vethed0ba5d from /proc [net=ffffffff88316180] [entry=ffff979406cda0c0]
[314162.745807] [PF_RING][DEBUG] ring_notifier:8994 veth361ac4f: [UNREGISTER][ifindex: 7744]
[314162.745820] [PF_RING][DEBUG] remove_device_from_ring_list:8698 removing dev=veth361ac4f ifindex=7744 (1)
[314162.745832] [PF_RING][DEBUG] remove_device_from_proc:8664 removing veth361ac4f/info from /proc [net=ffffffff88316180] [entry=ffff97972a9eec00]
[314162.745838] [PF_RING][DEBUG] remove_device_from_proc:8664 removing veth361ac4f/info from /proc [net=ffffffff88316180] [entry=ffff97972a9eec00]
[314162.745853] [PF_RING][DEBUG] remove_device_from_proc:8673 removing veth361ac4f from /proc [net=ffffffff88316180] [entry=ffff97972a9eeb40]
[314162.745855] ------------[ cut here ]------------
[314162.745944] WARNING: CPU: 3 PID: 11200 at fs/proc/generic.c:620 remove_proc_entry+0x18d/0x1a0
[314162.745950] name 'info'
[314162.745954] Modules linked in: pf_ring(OE-) veth vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 xt_statistic xt_nat nf_tables ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nfnetlink_queue xt_NFQUEUE xt_comment xt_mark iptable_mangle xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 sb_edac vfat fat iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev vmw_balloon ablk_helper joydev cryptd pcspkr sg nfit libnvdimm parport_pc vmw_vmci i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic vmwgfx pata_acpi drm_kms_helper syscopyarea sysfillrect
[314162.746150] sysimgblt fb_sys_fops ttm sd_mod ahci crc_t10dif crct10dif_generic libahci drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel serio_raw libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: pf_ring]
[314162.746201] CPU: 3 PID: 11200 Comm: rmmod Kdump: loaded Tainted: G OE ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[314162.746209] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/09/2019
[314162.746216] Call Trace:
[314162.746232] [