bugs
bugs copied to clipboard
Published
20 hours ago •
coreos
coreos
High kernel "SUnreclaimable" memory usage after around 5-7 days of uptime
Issue Report
Bug
The kernel's memory usage on a DigitalOcean's 1 GB droplet with CoreOS rises up to an unreasonable level after leaving the machine on for around 5-7 days, making it unresponsive and sometimes trigger OOM.
The machine is not running Kubernetes but is a (single host) Docker swarm manager. It has only 3 containers running.
Container Linux Version
$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=2247.6.0
VERSION_ID=2247.6.0
BUILD_ID=2019-11-06-2138
PRETTY_NAME="Container Linux by CoreOS 2247.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"
Environment
DigitalOcean cloud provider, Singapore POP, 1 GB machine.
Expected Behavior
The kernel uses a reasonable amount of memory. SClaimable may grow significantly, but SUnreclaimable shouldn't rise above 100 MB.
Actual Behavior
The kernel's unclaimable memory goes to almost 300 MB, while claimable memory stays around 57 MB. sudo slabtop --sort c shows that the top consumer is cred_jar at around 74 MB.
Reproduction Steps
- Start a new DigitalOcean 1GB droplet. Leave it running for about 5-7 days.
- Inspect kernel memory usage with
cat /proc/meminfoandsudo slabtop --sort c.
Other Information
This is the result of cat /proc/meminfo and sudo slabtop --sort c:
cat /proc/meminfo
MemTotal: 1009980 kB MemFree: 51916 kB MemAvailable: 13816 kB Buffers: 128 kB Cached: 235944 kB SwapCached: 0 kB Active: 400316 kB Inactive: 157704 kB Active(anon): 395984 kB Inactive(anon): 153584 kB Active(file): 4332 kB Inactive(file): 4120 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 321956 kB Mapped: 93824 kB Shmem: 227616 kB Slab: 357040 kB SReclaimable: 57848 kB SUnreclaim: 299192 kB KernelStack: 3200 kB PageTables: 5988 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 504988 kB Committed_AS: 2279580 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 3668 kB HardwareCorrupted: 0 kB AnonHugePages: 172032 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 417772 kB DirectMap2M: 630784 kB DirectMap1G: 0 kB
sudo slabtop --sort c
Active / Total Objects (% used) : 3054935 / 3075343 (99.3%)
Active / Total Slabs (% used) : 69313 / 69313 (100.0%)
Active / Total Caches (% used) : 101 / 130 (77.7%)
Active / Total Size (% used) : 345078.60K / 351209.45K (98.3%)
Minimum / Average / Maximum Object : 0.01K / 0.11K / 23.25K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
392847 392662 99% 0.19K 18707 21 74828K cred_jar
24855 24855 100% 1.06K 1657 15 26512K mm_struct
329856 329299 99% 0.06K 5154 64 20616K anon_vma_chain
32727 32389 98% 0.59K 2518 13 20144K inode_cache
79504 79362 99% 0.25K 4969 16 19876K filp
600064 600064 100% 0.03K 4688 128 18752K kmalloc-32
39360 39360 100% 0.38K 3936 10 15744K kmem_cache
7848 7836 99% 2.00K 981 8 15696K kmalloc-2048
159160 159160 100% 0.09K 3460 46 13840K anon_vma
18821 18811 99% 0.69K 1711 11 13688K sock_inode_cache
13448 13448 100% 1.00K 1681 8 13448K UNIX
96720 96720 100% 0.13K 3224 30 12896K kernfs_node_cache
58254 51782 88% 0.19K 2774 21 11096K dentry
2336 2334 99% 4.00K 292 8 9344K kmalloc-4096
15768 15764 99% 0.50K 1971 8 7884K kmalloc-512
53888 53888 100% 0.12K 1684 32 6736K pid
26816 26736 99% 0.25K 1676 16 6704K skbuff_head_cache
853504 853504 100% 0.01K 1667 512 6668K kmalloc-8
9506 7982 83% 0.57K 679 14 5432K radix_tree_node
5100 5099 99% 1.00K 638 8 5104K kmalloc-1024
4275 2390 55% 1.05K 285 15 4560K ext4_inode_cache
47292 47092 99% 0.09K 1126 42 4504K kmalloc-96
279 254 91% 9.25K 93 3 2976K task_struct
39488 39488 100% 0.06K 617 64 2468K kmem_cache_node
36416 36272 99% 0.06K 569 64 2276K kmalloc-64
7938 6512 82% 0.19K 378 21 1512K kmalloc-192
1705 1408 82% 0.69K 155 11 1240K shmem_inode_cache
10335 9796 94% 0.10K 265 39 1060K buffer_head
1404 1264 90% 0.66K 118 12 944K proc_inode_cache
3584 3584 100% 0.25K 224 16 896K kmalloc-256
21522 15903 73% 0.04K 211 102 844K numa_policy
96 75 78% 8.00K 24 4 768K kmalloc-8192
300 300 100% 2.06K 20 15 640K sighand_cache
5024 5024 100% 0.12K 157 32 628K kmalloc-128
19210 19210 100% 0.02K 113 170 452K avtab_node
390 390 100% 1.06K 26 15 416K signal_cache
52 34 65% 8.00K 13 4 416K biovec-max
26368 26368 100% 0.02K 103 256 412K kmalloc-16
588 415 70% 0.66K 49 12 392K ovl_inode
910 853 93% 0.38K 91 10 364K mnt_cache
4032 4032 100% 0.06K 63 64 252K ebitmap_node
98 98 100% 2.25K 7 14 224K TCPv6
1029 1029 100% 0.19K 49 21 196K proc_dir_entry
48 24 50% 4.00K 6 8 192K names_cache
242 242 100% 0.69K 22 11 176K files_cache
3485 3485 100% 0.05K 41 85 164K ftrace_event_field
75 75 100% 2.12K 5 15 160K TCP
3570 3162 88% 0.04K 35 102 140K ext4_extent_status
1380 1380 100% 0.09K 30 46 120K trace_event_file
870 870 100% 0.13K 29 30 116K dm_bufio_buffer-4
168 168 100% 0.62K 14 12 112K dio
98 98 100% 1.12K 7 14 112K RAWv6
1344 1344 100% 0.07K 24 56 96K Acpi-Operand
110 110 100% 0.75K 11 10 88K task_group
1120 1120 100% 0.07K 20 56 80K eventpoll_pwq
60 60 100% 1.25K 5 12 80K UDPv6
110 110 100% 0.69K 10 11 80K bio-2
216 170 78% 0.31K 18 12 72K nf_conntrack
204 156 76% 0.31K 17 12 68K xfrm_dst_cache
256 256 100% 0.25K 16 16 64K pool_workqueue
10 10 100% 5.50K 2 5 64K net_namespace
26 26 100% 2.40K 2 13 64K request_queue
3328 3328 100% 0.02K 13 256 52K selinux_file_security
416 416 100% 0.12K 13 32 52K scsi_sense_cache
408 408 100% 0.12K 12 34 48K jbd2_journal_head
1122 1122 100% 0.04K 11 102 44K pde_opener
88 88 100% 0.50K 11 8 44K skbuff_fclone_cache
231 231 100% 0.19K 11 21 44K dmaengine-unmap-16
187 187 100% 0.23K 11 17 44K tw_sock_TCP
187 187 100% 0.23K 11 17 44K tw_sock_TCPv6
40 40 100% 0.94K 5 8 40K RAW
459 459 100% 0.08K 9 51 36K inotify_inode_mark
38 27 71% 0.81K 2 19 32K bdev_cache
15 15 100% 2.06K 1 15 32K dmaengine-unmap-256
8 8 100% 4.00K 1 8 32K sgpool-128
91 91 100% 0.30K 7 13 28K blkdev_requests
196 196 100% 0.14K 7 28 28K ext4_groupinfo_4k
24 24 100% 1.00K 3 8 24K biovec-64
438 438 100% 0.05K 6 73 24K mbcache
15 15 100% 1.06K 1 15 16K dmaengine-unmap-128
8 8 100% 2.00K 1 8 16K biovec-128
52 52 100% 0.30K 4 13 16K request_sock_TCP
128 128 100% 0.12K 4 32 16K secpath_cache
512 512 100% 0.03K 4 128 16K fscrypt_info
22 22 100% 0.72K 2 11 16K fat_inode_cache
153 153 100% 0.08K 3 51 12K Acpi-State
219 219 100% 0.05K 3 73 12K Acpi-Parse
117 117 100% 0.10K 3 39 12K blkdev_ioc
192 192 100% 0.06K 3 64 12K ext4_io_end
24 24 100% 0.32K 2 12 8K taskstats
40 40 100% 0.20K 2 20 8K file_lock_cache
13 13 100% 0.61K 1 13 8K hugetlbfs_inode_cache
8 8 100% 0.94K 1 8 8K mqueue_inode_cache
9 9 100% 0.43K 1 9 4K uts_namespace
25 25 100% 0.16K 1 25 4K sigqueue
16 16 100% 0.25K 1 16 4K dquot
19 19 100% 0.20K 1 19 4K ip4-frags
13 13 100% 0.30K 1 13 4K request_sock_TCPv6
85 85 100% 0.05K 1 85 4K fscrypt_ctx
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
32 32 100% 0.12K 1 32 4K ext4_allocation_context
0 0 0% 0.09K 0 42 0K dma-kmalloc-96
0 0 0% 0.19K 0 21 0K dma-kmalloc-192
0 0 0% 0.01K 0 512 0K dma-kmalloc-8
0 0 0% 0.02K 0 256 0K dma-kmalloc-16
0 0 0% 0.03K 0 128 0K dma-kmalloc-32
0 0 0% 0.06K 0 64 0K dma-kmalloc-64
0 0 0% 0.12K 0 32 0K dma-kmalloc-128
0 0 0% 0.25K 0 16 0K dma-kmalloc-256
0 0 0% 0.50K 0 8 0K dma-kmalloc-512
0 0 0% 1.00K 0 8 0K dma-kmalloc-1024
0 0 0% 2.00K 0 8 0K dma-kmalloc-2048
0 0 0% 4.00K 0 8 0K dma-kmalloc-4096
0 0 0% 8.00K 0 4 0K dma-kmalloc-8192
0 0 0% 0.03K 0 128 0K avc_xperms_data
0 0 0% 0.12K 0 34 0K iint_cache
0 0 0% 0.45K 0 17 0K user_namespace
0 0 0% 0.94K 0 8 0K PING
0 0 0% 0.23K 0 17 0K posix_timers_cache
0 0 0% 0.03K 0 128 0K dnotify_struct
0 0 0% 0.12K 0 34 0K cfq_io_cq
0 0 0% 0.44K 0 9 0K bfq_queue
0 0 0% 1.12K 0 14 0K PINGv6
0 0 0% 2.57K 0 12 0K dm_uevent
0 0 0% 3.23K 0 9 0K kcopyd_job
0 0 0% 0.16K 0 24 0K kvm_mmu_page_header
0 0 0% 23.25K 0 1 0K kvm_vcpu
0 0 0% 0.13K 0 30 0K kvm_async_pf
0 0 0% 0.04K 0 102 0K fat_cache
0 0 0% 0.21K 0 18 0K nf_conntrack_expect
And this is kernel messages, captured about 6 hours later compared to the 2 logs above:
dmesg
[ 0.000000] Linux version 4.19.78-coreos (jenkins@ip-10-7-32-103) (gcc version 8.3.0 (Gentoo Hardened 8.3.0-r1 p1.1)) #1 SMP Wed Nov 6 20:58:31 -00 2019 [ 0.000000] Command line: BOOT_IMAGE=/coreos/vmlinuz-b mount.usr=/dev/mapper/usr verity.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 coreos.oem.id=digitalocean verity.usrhash=c36835d1bb91ebb00ca7b284c1148235c82498221011cc5f77614b883c441bb7 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64 [ 0.000000] x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64 [ 0.000000] x86/fpu: xstate_offset[5]: 960, xstate_sizes[5]: 64 [ 0.000000] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]: 512 [ 0.000000] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024 [ 0.000000] x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]: 8 [ 0.000000] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format. [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffafff] usable [ 0.000000] BIOS-e820: [mem 0x000000003fffb000-0x000000003fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.4 present. [ 0.000000] DMI: DigitalOcean Droplet, BIOS 20171212 12/12/2017 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 11e95001, primary cpu clock [ 0.000000] kvm-clock: using sched offset of 9959619446314041 cycles [ 0.000003] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000007] tsc: Detected 2294.608 MHz processor [ 0.001496] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.001499] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.001503] last_pfn = 0x3fffb max_arch_pfn = 0x400000000 [ 0.001559] MTRR default type: write-back [ 0.001560] MTRR fixed ranges enabled: [ 0.001561] 00000-9FFFF write-back [ 0.001562] A0000-BFFFF uncachable [ 0.001562] C0000-FFFFF write-protect [ 0.001563] MTRR variable ranges enabled: [ 0.001565] 0 base 0080000000 mask FF80000000 uncachable [ 0.001565] 1 disabled [ 0.001566] 2 disabled [ 0.001566] 3 disabled [ 0.001567] 4 disabled [ 0.001567] 5 disabled [ 0.001568] 6 disabled [ 0.001568] 7 disabled [ 0.001583] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.001743] Using GB pages for direct mapping [ 0.001749] BRK [0x12001000, 0x12001fff] PGTABLE [ 0.001750] BRK [0x12002000, 0x12002fff] PGTABLE [ 0.001751] BRK [0x12003000, 0x12003fff] PGTABLE [ 0.001778] BRK [0x12004000, 0x12004fff] PGTABLE [ 0.001835] BRK [0x12005000, 0x12005fff] PGTABLE [ 0.001871] ACPI: Early table checksum verification disabled [ 0.001925] ACPI: RSDP 0x00000000000F1080 000014 (v00 BOCHS ) [ 0.001932] ACPI: RSDT 0x000000003FFFE400 000034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.001940] ACPI: FACP 0x000000003FFFFF80 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.001946] ACPI: DSDT 0x000000003FFFE440 001189 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.001950] ACPI: FACS 0x000000003FFFFF40 000040 [ 0.001953] ACPI: SSDT 0x000000003FFFF700 00083B (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.001957] ACPI: APIC 0x000000003FFFF610 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.001960] ACPI: HPET 0x000000003FFFF5D0 000038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) [ 0.001972] ACPI: Local APIC address 0xfee00000 [ 0.002513] No NUMA configuration found [ 0.002515] Faking a node at [mem 0x0000000000000000-0x000000003fffafff] [ 0.002521] NODE_DATA(0) allocated [mem 0x3fff5000-0x3fffafff] [ 0.002556] Zone ranges: [ 0.002557] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.002558] DMA32 [mem 0x0000000001000000-0x000000003fffafff] [ 0.002559] Normal empty [ 0.002560] Movable zone start for each node [ 0.002561] Early memory node ranges [ 0.002562] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.002563] node 0: [mem 0x0000000000100000-0x000000003fffafff] [ 0.002566] Reserved but unavailable: 98 pages [ 0.002567] Initmem setup node 0 [mem 0x0000000000001000-0x000000003fffafff] [ 0.002568] On node 0 totalpages: 262041 [ 0.002569] DMA zone: 64 pages used for memmap [ 0.002570] DMA zone: 21 pages reserved [ 0.002570] DMA zone: 3998 pages, LIFO batch:0 [ 0.002655] DMA32 zone: 4032 pages used for memmap [ 0.002655] DMA32 zone: 258043 pages, LIFO batch:63 [ 0.008884] ACPI: PM-Timer IO Port: 0xb008 [ 0.008889] ACPI: Local APIC address 0xfee00000 [ 0.008901] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.008945] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.008948] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.008950] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.008951] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.008960] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.008960] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.008962] ACPI: IRQ0 used by override. [ 0.008963] ACPI: IRQ5 used by override. [ 0.008963] ACPI: IRQ9 used by override. [ 0.008964] ACPI: IRQ10 used by override. [ 0.008964] ACPI: IRQ11 used by override. [ 0.008966] Using ACPI (MADT) for SMP configuration information [ 0.008968] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.008977] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.008997] [mem 0x40000000-0xfeffbfff] available for PCI devices [ 0.008998] Booting paravirtualized kernel on KVM [ 0.009001] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.105550] random: get_random_bytes called from start_kernel+0x93/0x522 with crng_init=0 [ 0.105562] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:1 nr_node_ids:1 [ 0.105935] percpu: Embedded 44 pages/cpu s143256 r8192 d28776 u2097152 [ 0.105943] pcpu-alloc: s143256 r8192 d28776 u2097152 alloc=1*2097152 [ 0.105944] pcpu-alloc: [0] 0 [ 0.105965] KVM setup async PF for cpu 0 [ 0.105971] kvm-stealtime: cpu 0, msr 3ea161c0 [ 0.105978] Built 1 zonelists, mobility grouping on. Total pages: 257924 [ 0.105979] Policy zone: DMA32 [ 0.105982] Kernel command line: rootflags=rw mount.usrflags=ro BOOT_IMAGE=/coreos/vmlinuz-b mount.usr=/dev/mapper/usr verity.usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 coreos.oem.id=digitalocean verity.usrhash=c36835d1bb91ebb00ca7b284c1148235c82498221011cc5f77614b883c441bb7 [ 0.109519] Memory: 965792K/1048164K available (10252K kernel code, 1217K rwdata, 5484K rodata, 41484K init, 1572K bss, 82372K reserved, 0K cma-reserved) [ 0.109641] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.109643] Kernel/User page tables isolation: enabled [ 0.109751] ftrace: allocating 29193 entries in 115 pages [ 0.122904] rcu: Hierarchical RCU implementation. [ 0.122908] rcu: RCU event tracing is enabled. [ 0.122909] rcu: RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=1. [ 0.122911] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 [ 0.125667] NR_IRQS: 33024, nr_irqs: 256, preallocated irqs: 16 [ 0.140423] Console: colour VGA+ 80x25 [ 0.217930] console [tty0] enabled [ 0.498626] console [ttyS0] enabled [ 0.501707] ACPI: Core revision 20180810 [ 0.504789] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.513928] hpet clockevent registered [ 0.514017] APIC: Switch to symmetric I/O mode setup [ 0.520743] x2apic enabled [ 0.524782] Switched APIC routing to physical x2apic. [ 0.532825] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.537414] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x21134f58f0d, max_idle_ns: 440795217993 ns [ 0.545831] Calibrating delay loop (skipped) preset value.. 4589.21 BogoMIPS (lpj=2294608) [ 0.546820] pid_max: default: 32768 minimum: 301 [ 0.548858] Security Framework initialized [ 0.549811] SELinux: Initializing. [ 0.552016] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.552930] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.553841] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.554824] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.557134] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8 [ 0.557812] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4 [ 0.558818] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization [ 0.559830] Spectre V2 : Mitigation: Full generic retpoline [ 0.560820] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch [ 0.561819] Spectre V2 : Enabling Restricted Speculation for firmware calls [ 0.562835] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier [ 0.563818] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.564866] MDS: Mitigation: Clear CPU buffers [ 0.575409] Freeing SMP alternatives memory: 28K [ 0.578805] TSC deadline timer enabled [ 0.578805] smpboot: CPU0: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (family: 0x6, model: 0x55, stepping: 0x4) [ 0.579110] Performance Events: Skylake events, Intel PMU driver. [ 0.579832] ... version: 2 [ 0.580823] ... bit width: 48 [ 0.581818] ... generic registers: 4 [ 0.582820] ... value mask: 0000ffffffffffff [ 0.583818] ... max period: 000000007fffffff [ 0.584816] ... fixed-purpose events: 3 [ 0.585820] ... event mask: 000000070000000f [ 0.586934] rcu: Hierarchical SRCU implementation. [ 0.588917] smp: Bringing up secondary CPUs ... [ 0.589831] smp: Brought up 1 node, 1 CPU [ 0.590822] smpboot: Max logical packages: 1 [ 0.591823] smpboot: Total of 1 processors activated (4589.21 BogoMIPS) [ 0.593178] devtmpfs: initialized [ 0.593916] x86/mm: Memory block size: 128MB [ 0.595230] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.595829] futex hash table entries: 256 (order: 2, 16384 bytes) [ 0.596930] pinctrl core: initialized pinctrl subsystem [ 0.597960] NET: Registered protocol family 16 [ 0.598957] audit: initializing netlink subsys (disabled) [ 0.599975] cpuidle: using governor menu [ 0.600945] ACPI: bus type PCI registered [ 0.601827] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.602914] dca service started, version 1.12.1 [ 0.603930] PCI: Using configuration type 1 for base access [ 0.604882] audit: type=2000 audit(1573260144.515:1): state=initialized audit_enabled=0 res=1 [ 0.606824] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.607818] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.609164] ACPI: Added _OSI(Module Device) [ 0.609827] ACPI: Added _OSI(Processor Device) [ 0.610824] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.611823] ACPI: Added _OSI(Processor Aggregator Device) [ 0.612822] ACPI: Added _OSI(Linux-Dell-Video) [ 0.613819] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio) [ 0.616310] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.618143] ACPI: Interpreter enabled [ 0.618850] ACPI: (supports S0 S3 S5) [ 0.619822] ACPI: Using IOAPIC for interrupt routing [ 0.620861] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.622245] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.625325] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.625832] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.626860] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.627849] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.629476] acpiphp: Slot [3] registered [ 0.629913] acpiphp: Slot [4] registered [ 0.630892] acpiphp: Slot [5] registered [ 0.631931] acpiphp: Slot [6] registered [ 0.632938] acpiphp: Slot [7] registered [ 0.633989] acpiphp: Slot [8] registered [ 0.634994] acpiphp: Slot [9] registered [ 0.636025] acpiphp: Slot [10] registered [ 0.636948] acpiphp: Slot [11] registered [ 0.637914] acpiphp: Slot [12] registered [ 0.638901] acpiphp: Slot [13] registered [ 0.639912] acpiphp: Slot [14] registered [ 0.640971] acpiphp: Slot [15] registered [ 0.641904] acpiphp: Slot [16] registered [ 0.642886] acpiphp: Slot [17] registered [ 0.643874] acpiphp: Slot [18] registered [ 0.644872] acpiphp: Slot [19] registered [ 0.645991] acpiphp: Slot [20] registered [ 0.646916] acpiphp: Slot [21] registered [ 0.647919] acpiphp: Slot [22] registered [ 0.648912] acpiphp: Slot [23] registered [ 0.649911] acpiphp: Slot [24] registered [ 0.650891] acpiphp: Slot [25] registered [ 0.651888] acpiphp: Slot [26] registered [ 0.652865] acpiphp: Slot [27] registered [ 0.653875] acpiphp: Slot [28] registered [ 0.654903] acpiphp: Slot [29] registered [ 0.655925] acpiphp: Slot [30] registered [ 0.656898] acpiphp: Slot [31] registered [ 0.657851] PCI host bridge to bus 0000:00 [ 0.658824] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.659824] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.660824] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.661823] pci_bus 0000:00: root bus resource [mem 0x80000000-0xfebfffff window] [ 0.662824] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.663900] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.664705] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.665762] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.668131] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.669908] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.670822] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.671822] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.672822] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.674169] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300 [ 0.677488] pci 0000:00:01.2: reg 0x20: [io 0xc080-0xc09f] [ 0.678472] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.679183] pci 0000:00:01.3: quirk: [io 0xb000-0xb03f] claimed by PIIX4 ACPI [ 0.679845] pci 0000:00:01.3: quirk: [io 0xb100-0xb10f] claimed by PIIX4 SMB [ 0.681538] pci 0000:00:02.0: [1b36:0100] type 00 class 0x030000 [ 0.682866] pci 0000:00:02.0: reg 0x10: [mem 0xf8000000-0xfbffffff] [ 0.684868] pci 0000:00:02.0: reg 0x14: [mem 0xfc000000-0xfcffffff] [ 0.686899] pci 0000:00:02.0: reg 0x18: [mem 0xfd010000-0xfd011fff] [ 0.688902] pci 0000:00:02.0: reg 0x1c: [io 0xc0a0-0xc0bf] [ 0.694872] pci 0000:00:02.0: reg 0x30: [mem 0xfd000000-0xfd00ffff pref] [ 0.696111] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 [ 0.696818] pci 0000:00:03.0: reg 0x10: [io 0xc0c0-0xc0df] [ 0.697689] pci 0000:00:03.0: reg 0x14: [mem 0xfd012000-0xfd012fff] [ 0.703060] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 [ 0.703818] pci 0000:00:04.0: reg 0x10: [io 0xc0e0-0xc0ff] [ 0.705366] pci 0000:00:04.0: reg 0x14: [mem 0xfd013000-0xfd013fff] [ 0.711491] pci 0000:00:05.0: [1af4:1004] type 00 class 0x010000 [ 0.712309] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc03f] [ 0.713244] pci 0000:00:05.0: reg 0x14: [mem 0xfd014000-0xfd014fff] [ 0.719516] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.720352] pci 0000:00:06.0: reg 0x10: [io 0xc040-0xc07f] [ 0.721350] pci 0000:00:06.0: reg 0x14: [mem 0xfd015000-0xfd015fff] [ 0.728112] pci 0000:00:07.0: [1af4:1002] type 00 class 0x00ff00 [ 0.728754] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc11f] [ 0.732680] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.733936] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.735062] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.736109] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.737012] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.738402] pci 0000:00:02.0: vgaarb: setting as boot VGA device [ 0.738805] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none [ 0.738825] pci 0000:00:02.0: vgaarb: bridge control possible [ 0.739819] vgaarb: loaded [ 0.740943] pps_core: LinuxPPS API ver. 1 registered [ 0.741818] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti[ 0.742841] PTP clock support registered [ 0.743917] PCI: Using ACPI for IRQ routing [ 0.744822] PCI: pci_cache_line_size set to 64 bytes [ 0.745112] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.745118] e820: reserve RAM buffer [mem 0x3fffb000-0x3fffffff] [ 0.745693] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 0.745884] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.746824] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.751892] clocksource: Switched to clocksource kvm-clock [ 0.768793] VFS: Disk quotas dquot_6.6.0 [ 0.771205] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.774890] pnp: PnP ACPI init [ 0.777241] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.777267] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.777283] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.777302] pnp 00:03: [dma 2] [ 0.777310] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.777388] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.777599] pnp: PnP ACPI: found 5 devices [ 0.788030] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.796986] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.796988] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.796989] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.796991] pci_bus 0000:00: resource 7 [mem 0x80000000-0xfebfffff window] [ 0.797116] NET: Registered protocol family 2 [ 0.800605] tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 8192 bytes) [ 0.806250] TCP established hash table entries: 8192 (order: 4, 65536 bytes) [ 0.810450] TCP bind hash table entries: 8192 (order: 5, 131072 bytes) [ 0.814563] TCP: Hash tables configured (established 8192 bind 8192) [ 0.818494] UDP hash table entries: 512 (order: 2, 16384 bytes) [ 0.821941] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes) [ 0.827063] NET: Registered protocol family 1 [ 0.829882] NET: Registered protocol family 44 [ 0.832675] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.836400] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.840057] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.843680] PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 0.847649] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff] [ 0.853296] PCI: CLS 0 bytes, default 64 [ 1.496143] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x21134f58f0d, max_idle_ns: 440795217993 ns [ 1.503844] Initialise system trusted keyrings [ 1.508124] workingset: timestamp_bits=39 max_order=18 bucket_order=0 [ 1.758294] Key type asymmetric registered [ 1.761449] Asymmetric key parser 'x509' registered [ 1.766939] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 1.774415] io scheduler noop registered [ 1.778037] io scheduler deadline registered [ 1.781382] io scheduler cfq registered (default) [ 1.784513] io scheduler mq-deadline registered [ 1.787680] io scheduler kyber registered [ 1.792388] io scheduler bfq registered [ 1.795236] intel_idle: Please enable MWAIT in BIOS SETUP [ 1.795476] ioatdma: Intel(R) QuickData Technology Driver 4.00 [ 1.799595] PCI Interrupt Link [LNKC] enabled at IRQ 10 [ 1.802221] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 1.806042] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver [ 1.810017] PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.812152] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 1.816254] PCI Interrupt Link [LNKB] enabled at IRQ 11 [ 1.822850] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver [ 1.829917] virtio-pci 0000:00:07.0: virtio_pci: leaving for legacy driver [ 1.834834] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 1.863736] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 1.869110] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.874991] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.877521] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.880972] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 [ 1.888544] rtc_cmos 00:00: RTC can wake from S4 [ 1.891674] rtc_cmos 00:00: registered as rtc0 [ 1.894056] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram, hpet irqs [ 1.897523] NET: Registered protocol family 10 [ 1.900680] Segment Routing with IPv6 [ 1.903157] NET: Registered protocol family 17 [ 1.906120] Key type dns_resolver registered [ 1.908924] mce: Using 10 MCE banks [ 1.911612] sched_clock: Marking stable (1497656423, 413877026)->(2374759917, -463226468) [ 1.919693] registered taskstats version 1 [ 1.922651] Loading compiled-in X.509 certificates [ 1.961181] Loaded X.509 cert 'CoreOS, Inc: Module signing key for 4.19.78-coreos: 317d2ce0a4b0e4c89f765793a811eec6a79f49c3' [ 1.968101] ima: No TPM chip found, activating TPM-bypass! [ 1.971482] ima: Allocated hash algorithm: sha1 [ 1.974601] rtc_cmos 00:00: setting system clock to 2019-11-09 00:42:25 UTC (1573260145) [ 1.991636] Freeing unused kernel image memory: 41484K [ 1.995869] Write protecting the kernel read-only data: 18432k [ 2.001096] Freeing unused kernel image memory: 2016K [ 2.004846] Freeing unused kernel image memory: 660K [ 2.009673] Run /init as init process [ 2.020731] systemd[1]: systemd 241-30-gf0da8f7+ running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.037372] systemd[1]: Detected virtualization kvm. [ 2.041044] systemd[1]: Detected architecture x86-64. [ 2.044064] systemd[1]: Running in initial RAM disk. [ 2.055325] systemd[1]: No hostname configured. [ 2.057911] systemd[1]: Set hostname to . [ 2.061098] systemd[1]: Initializing machine ID from KVM UUID. [ 2.065217] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument [ 2.210451] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling. [ 2.225098] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.) [ 2.517723] device-mapper: uevent: version 1.0.3 [ 2.525996] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: [email protected] [ 3.029192] audit: type=1130 audit(1573260146.554:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3.184026] audit: type=1130 audit(1573260146.709:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udevd comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3.339165] audit: type=1130 audit(1573260146.864:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3.505341] audit: type=1130 audit(1573260147.030:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 3.736327] virtio_blk virtio3: [vda] 52428800 512-byte logical blocks (26.8 GB/25.0 GiB) [ 3.829074] cryptd: max_cpu_qlen set to 1000 [ 3.846092] SCSI subsystem initialized [ 3.861043] ACPI: bus type USB registered [ 3.870092] scsi host0: Virtio SCSI HBA [ 3.890726] vda: vda1 vda2 vda3 vda4 vda6 vda7 vda9 [ 3.945008] usbcore: registered new interface driver usbfs [ 4.008157] audit: type=1130 audit(1573260147.533:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-networkd comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 4.025057] random: fast init done [ 4.025703] usbcore: registered new interface driver hub [ 4.025897] usbcore: registered new device driver usb [ 4.059473] AVX2 version of gcm_enc/dec engaged. [ 4.059474] AES CTR mode by8 optimization enabled [ 4.082058] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 4.092444] libata version 3.00 loaded. [ 4.095312] ehci-pci: EHCI PCI platform driver [ 4.119207] ata_piix 0000:00:01.1: version 2.13 [ 4.126515] uhci_hcd: USB Universal Host Controller Interface driver [ 4.141319] scsi host1: ata_piix [ 4.170648] scsi host2: ata_piix [ 4.218017] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.222520] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 4.242356] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 4.279387] random: veritysetup: uninitialized urandom read (2 bytes read) [ 4.326362] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 4.326434] uhci_hcd 0000:00:01.2: detected 2 ports [ 4.326629] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c080 [ 4.346338] hub 1-0:1.0: USB hub found [ 4.346348] hub 1-0:1.0: 2 ports detected [ 4.417296] A link change request failed with some changes committed already. Interface eth1 may have been left with an inconsistent configuration, please check. [ 4.426257] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.426262] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check. [ 4.712728] audit: type=1130 audit(1573260148.237:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 4.736898] audit: type=1130 audit(1573260148.255:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=verity-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 4.815049] audit: type=1130 audit(1573260148.340:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-mount comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 4.859849] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 4.888709] audit: type=1130 audit(1573260148.413:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 4.979799] EXT4-fs (vda9): mounted filesystem with ordered data mode. Opts: (null) [ 5.036074] random: coreos-metadata: uninitialized urandom read (16 bytes read) [ 5.046906] random: reqwest-interna: uninitialized urandom read (32 bytes read) [ 5.097371] EXT4-fs (dm-0): mounted filesystem without journal. Opts: (null) [ 12.128019] kauditd_printk_skb: 4 callbacks suppressed [ 12.128020] audit: type=1130 audit(1573260155.653:15): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=coreos-digitalocean-network comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.186689] audit: type=1131 audit(1573260155.671:16): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=coreos-digitalocean-network comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.208656] audit: type=1130 audit(1573260155.733:17): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.264469] audit: type=1131 audit(1573260155.787:18): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-pivot comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.297936] audit: type=1131 audit(1573260155.823:19): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-mount comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.311244] audit: type=1131 audit(1573260155.836:20): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.390163] audit: type=1131 audit(1573260155.915:21): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-udev-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.423312] audit: type=1131 audit(1573260155.948:22): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-pre-trigger comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.466490] audit: type=1130 audit(1573260155.986:23): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-cleanup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 12.466493] audit: type=1131 audit(1573260155.986:24): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-cleanup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 13.145934] systemd-journald[143]: Received SIGTERM from PID 1 (n/a). [ 13.193781] systemd: 28 output lines suppressed due to ratelimiting [ 13.363971] SELinux: Permission validate_trans in class security not defined in policy. [ 13.370003] SELinux: Permission getrlimit in class process not defined in policy. [ 13.376609] SELinux: Class process2 not defined in policy. [ 13.381031] SELinux: Permission module_load in class system not defined in policy. [ 13.386606] SELinux: Permission map in class file not defined in policy. [ 13.392250] SELinux: Permission map in class dir not defined in policy. [ 13.396583] SELinux: Permission map in class lnk_file not defined in policy. [ 13.400174] SELinux: Permission map in class chr_file not defined in policy. [ 13.403992] SELinux: Permission map in class blk_file not defined in policy. [ 13.407838] SELinux: Permission map in class sock_file not defined in policy. [ 13.412735] SELinux: Permission map in class fifo_file not defined in policy. [ 13.418828] SELinux: Permission map in class socket not defined in policy. [ 13.426682] SELinux: Permission map in class tcp_socket not defined in policy. [ 13.433456] SELinux: Permission map in class udp_socket not defined in policy. [ 13.439723] SELinux: Permission map in class rawip_socket not defined in policy. [ 13.448026] SELinux: Permission map in class netlink_socket not defined in policy. [ 13.457157] SELinux: Permission map in class packet_socket not defined in policy. [ 13.462436] SELinux: Permission map in class key_socket not defined in policy. [ 13.468121] SELinux: Permission map in class unix_stream_socket not defined in policy. [ 13.473096] SELinux: Permission map in class unix_dgram_socket not defined in policy. [ 13.478628] SELinux: Permission map in class netlink_route_socket not defined in policy. [ 13.485004] SELinux: Permission map in class netlink_tcpdiag_socket not defined in policy. [ 13.488469] SELinux: Permission map in class netlink_nflog_socket not defined in policy. [ 13.493001] SELinux: Permission map in class netlink_xfrm_socket not defined in policy. [ 13.497794] SELinux: Permission map in class netlink_selinux_socket not defined in policy. [ 13.503468] SELinux: Permission map in class netlink_iscsi_socket not defined in policy. [ 13.509465] SELinux: Permission map in class netlink_audit_socket not defined in policy. [ 13.517472] SELinux: Permission map in class netlink_fib_lookup_socket not defined in policy. [ 13.524126] SELinux: Permission map in class netlink_connector_socket not defined in policy. [ 13.529969] SELinux: Permission map in class netlink_netfilter_socket not defined in policy. [ 13.534701] SELinux: Permission map in class netlink_dnrt_socket not defined in policy. [ 13.539544] SELinux: Permission map in class netlink_kobject_uevent_socket not defined in policy. [ 13.549639] SELinux: Permission map in class netlink_generic_socket not defined in policy. [ 13.555971] SELinux: Permission map in class netlink_scsitransport_socket not defined in policy. [ 13.562031] SELinux: Permission map in class netlink_rdma_socket not defined in policy. [ 13.567434] SELinux: Permission map in class netlink_crypto_socket not defined in policy. [ 13.572056] SELinux: Permission map in class appletalk_socket not defined in policy. [ 13.576993] SELinux: Permission map in class dccp_socket not defined in policy. [ 13.583345] SELinux: Permission map in class tun_socket not defined in policy. [ 13.588902] SELinux: Class cap_userns not defined in policy. [ 13.592825] SELinux: Class cap2_userns not defined in policy. [ 13.596205] SELinux: Class sctp_socket not defined in policy. [ 13.600050] SELinux: Class icmp_socket not defined in policy. [ 13.603712] SELinux: Class ax25_socket not defined in policy. [ 13.607356] SELinux: Class ipx_socket not defined in policy. [ 13.613402] SELinux: Class netrom_socket not defined in policy. [ 13.618219] SELinux: Class atmpvc_socket not defined in policy. [ 13.622539] SELinux: Class x25_socket not defined in policy. [ 13.626643] SELinux: Class rose_socket not defined in policy. [ 13.630095] SELinux: Class decnet_socket not defined in policy. [ 13.633622] SELinux: Class atmsvc_socket not defined in policy. [ 13.636927] SELinux: Class rds_socket not defined in policy. [ 13.642280] SELinux: Class irda_socket not defined in policy. [ 13.646084] SELinux: Class pppox_socket not defined in policy. [ 13.649855] SELinux: Class llc_socket not defined in policy. [ 13.653445] SELinux: Class can_socket not defined in policy. [ 13.657673] SELinux: Class tipc_socket not defined in policy. [ 13.663193] SELinux: Class bluetooth_socket not defined in policy. [ 13.668370] SELinux: Class iucv_socket not defined in policy. [ 13.673870] SELinux: Class rxrpc_socket not defined in policy. [ 13.677235] SELinux: Class isdn_socket not defined in policy. [ 13.680005] SELinux: Class phonet_socket not defined in policy. [ 13.683483] SELinux: Class ieee802154_socket not defined in policy. [ 13.686523] SELinux: Class caif_socket not defined in policy. [ 13.689456] SELinux: Class alg_socket not defined in policy. [ 13.692905] SELinux: Class nfc_socket not defined in policy. [ 13.696133] SELinux: Class vsock_socket not defined in policy. [ 13.700049] SELinux: Class kcm_socket not defined in policy. [ 13.705354] SELinux: Class qipcrtr_socket not defined in policy. [ 13.708121] SELinux: Class smc_socket not defined in policy. [ 13.711840] SELinux: Class infiniband_pkey not defined in policy. [ 13.716033] SELinux: Class infiniband_endport not defined in policy. [ 13.719201] SELinux: Class bpf not defined in policy. [ 13.722055] SELinux: Class xdp_socket not defined in policy. [ 13.725529] SELinux: the above unknown classes and permissions will be allowed [ 13.730375] SELinux: policy capability network_peer_controls=1 [ 13.735578] SELinux: policy capability open_perms=1 [ 13.738300] SELinux: policy capability extended_socket_class=0 [ 13.741949] SELinux: policy capability always_check_network=0 [ 13.745520] SELinux: policy capability cgroup_seclabel=0 [ 13.748784] SELinux: policy capability nnp_nosuid_transition=0 [ 13.773588] systemd[1]: Successfully loaded SELinux policy in 451.966ms. [ 13.802321] systemd[1]: Relabelled /dev, /dev/shm, /run, /sys/fs/cgroup in 9.597ms. [ 13.809599] systemd[1]: systemd 241-30-gf0da8f7+ running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.824758] systemd[1]: Detected virtualization kvm. [ 13.828691] systemd[1]: Detected architecture x86-64. [ 13.836896] systemd[1]: Set hostname to . [ 13.845542] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument [ 13.977078] urandom_read: 1 callbacks suppressed [ 13.977080] random: lvm2-activation: uninitialized urandom read (4 bytes read) [ 17.156225] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling. [ 17.171667] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.) [ 17.344590] kauditd_printk_skb: 10 callbacks suppressed [ 17.344591] audit: type=1131 audit(1573260160.869:35): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.374429] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.379784] systemd[1]: Stopped Switch Root. [ 17.387356] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.405850] audit: type=1130 audit(1573260160.911:36): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.475376] audit: type=1131 audit(1573260160.911:37): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.475381] audit: type=1130 audit(1573260160.921:38): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.475384] audit: type=1131 audit(1573260160.922:39): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.664828] audit: type=1131 audit(1573260161.189:40): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-fsck-root comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.816025] random: systemd-random-: uninitialized urandom read (512 bytes read) [ 17.854055] audit: type=1131 audit(1573260161.379:41): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=verity-setup comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.935020] audit: type=1130 audit(1573260161.460:42): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=coreos-tmpfiles comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.974899] audit: type=1131 audit(1573260161.474:43): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=coreos-tmpfiles comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 17.974903] audit: type=1130 audit(1573260161.481:44): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:kernel_t:s0 msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib64/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 18.424338] systemd-journald[510]: Received request to flush runtime journal from PID 1 [ 19.399277] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 19.483828] ACPI: Power Button [PWRF] [ 19.585421] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 [ 19.625712] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 [ 19.956102] A link change request failed with some changes committed already. Interface eth1 may have been left with an inconsistent configuration, please check. [ 19.987040] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check. [ 20.188096] mousedev: PS/2 mouse device common for all mice [ 20.368375] random: lvm: uninitialized urandom read (4 bytes read) [ 20.430325] random: lvm: uninitialized urandom read (4 bytes read) [ 20.559995] EXT4-fs (vda6): mounted filesystem with ordered data mode. Opts: (null) [ 21.310707] random: mktemp: uninitialized urandom read (8 bytes read) [ 22.203072] random: dbus-daemon: uninitialized urandom read (12 bytes read) [ 29.844292] random: crng init done [ 33.926309] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 33.935859] Bridge firewalling registered [ 34.051014] Initializing XFRM netlink socket [ 34.177247] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready [ 34.253042] IPv6: ADDRCONF(NETDEV_UP): docker_gwbridge: link is not ready [ 36.621177] br0: renamed from ov-001000-g9bon [ 36.628624] vxlan0: renamed from vx-001000-g9bon [ 36.632668] br0: port 1(vxlan0) entered blocking state [ 36.635654] br0: port 1(vxlan0) entered disabled state [ 36.640618] device vxlan0 entered promiscuous mode [ 36.650943] br0: port 1(vxlan0) entered blocking state [ 36.654855] br0: port 1(vxlan0) entered forwarding state [ 36.746749] veth0: renamed from vetheb536de [ 36.753988] br0: port 2(veth0) entered blocking state [ 36.756432] br0: port 2(veth0) entered disabled state [ 36.765126] device veth0 entered promiscuous mode [ 36.772951] br0: port 2(veth0) entered blocking state [ 36.778320] br0: port 2(veth0) entered forwarding state [ 37.061016] eth0: renamed from veth0ea39b3 [ 37.194076] docker_gwbridge: port 1(veth9c641ee) entered blocking state [ 37.199509] docker_gwbridge: port 1(veth9c641ee) entered disabled state [ 37.218520] device veth9c641ee entered promiscuous mode [ 37.239889] IPv6: ADDRCONF(NETDEV_UP): veth9c641ee: link is not ready [ 37.245582] docker_gwbridge: port 1(veth9c641ee) entered blocking state [ 37.252445] docker_gwbridge: port 1(veth9c641ee) entered forwarding state [ 37.257112] docker_gwbridge: port 1(veth9c641ee) entered disabled state [ 37.391026] eth1: renamed from vethe0c236d [ 37.396135] IPv6: ADDRCONF(NETDEV_CHANGE): veth9c641ee: link becomes ready [ 37.400294] docker_gwbridge: port 1(veth9c641ee) entered blocking state [ 37.403902] docker_gwbridge: port 1(veth9c641ee) entered forwarding state [ 37.417943] IPv6: ADDRCONF(NETDEV_CHANGE): docker_gwbridge: link becomes ready [ 37.522541] br0: renamed from ov-001002-0suyd [ 37.534524] vxlan0: renamed from vx-001002-0suyd [ 37.544607] br0: port 1(vxlan0) entered blocking state [ 37.549367] br0: port 1(vxlan0) entered disabled state [ 37.566940] device vxlan0 entered promiscuous mode [ 37.579990] br0: port 1(vxlan0) entered blocking state [ 37.584287] br0: port 1(vxlan0) entered forwarding state [ 37.617613] br0: renamed from ov-001001-ka7zk [ 37.628040] vxlan0: renamed from vx-001001-ka7zk [ 37.636606] br0: port 1(vxlan0) entered blocking state [ 37.640146] br0: port 1(vxlan0) entered disabled state [ 37.657942] device vxlan0 entered promiscuous mode [ 37.667112] br0: port 1(vxlan0) entered blocking state [ 37.670252] br0: port 1(vxlan0) entered forwarding state [ 37.756629] veth0: renamed from vethbe15515 [ 37.766331] br0: port 2(veth0) entered blocking state [ 37.769654] br0: port 2(veth0) entered disabled state [ 37.794938] device veth0 entered promiscuous mode [ 37.827900] br0: port 2(veth0) entered blocking state [ 37.830899] br0: port 2(veth0) entered forwarding state [ 37.872086] veth0: renamed from vetha93d670 [ 37.885031] br0: port 2(veth0) entered blocking state [ 37.889036] br0: port 2(veth0) entered disabled state [ 37.913935] device veth0 entered promiscuous mode [ 37.942964] br0: port 2(veth0) entered blocking state [ 37.946090] br0: port 2(veth0) entered forwarding state [ 38.069475] eth0: renamed from vethc42dadf [ 38.074029] br0: port 2(veth0) entered disabled state [ 38.118441] eth0: renamed from veth3dae14a [ 38.125021] br0: port 2(veth0) entered blocking state [ 38.128319] br0: port 2(veth0) entered forwarding state [ 38.807001] br0: port 2(veth0) entered disabled state [ 38.809246] br0: port 1(vxlan0) entered disabled state [ 38.813904] ov-001001-ka7zk: renamed from br0 [ 38.816057] device veth0 left promiscuous mode [ 38.818236] ov-001001-ka7zk: port 2(veth0) entered disabled state [ 38.821375] device vxlan0 left promiscuous mode [ 38.823051] ov-001001-ka7zk: port 1(vxlan0) entered disabled state [ 38.829154] vx-001001-ka7zk: renamed from vxlan0 [ 38.838113] vetha93d670: renamed from veth0 [ 38.863070] veth3dae14a: renamed from eth0 [ 38.868010] br0: port 2(veth0) entered disabled state [ 38.871258] br0: port 1(vxlan0) entered disabled state [ 38.884283] ov-001002-0suyd: renamed from br0 [ 38.889939] device veth0 left promiscuous mode [ 38.892652] ov-001002-0suyd: port 2(veth0) entered disabled state [ 38.902098] device vxlan0 left promiscuous mode [ 38.905378] ov-001002-0suyd: port 1(vxlan0) entered disabled state [ 38.930911] vx-001002-0suyd: renamed from vxlan0 [ 38.951560] vethbe15515: renamed from veth0 [ 38.967096] vethc42dadf: renamed from eth0 [ 42.062319] br0: renamed from ov-001002-0suyd [ 42.070555] vxlan0: renamed from vx-001002-0suyd [ 42.077711] br0: port 1(vxlan0) entered blocking state [ 42.079978] br0: port 1(vxlan0) entered disabled state [ 42.089902] device vxlan0 entered promiscuous mode [ 42.098943] br0: port 1(vxlan0) entered blocking state [ 42.101684] br0: port 1(vxlan0) entered forwarding state [ 42.134021] br0: renamed from ov-001001-ka7zk [ 42.142550] vxlan0: renamed from vx-001001-ka7zk [ 42.148479] br0: port 1(vxlan0) entered blocking state [ 42.151851] br0: port 1(vxlan0) entered disabled state [ 42.170927] device vxlan0 entered promiscuous mode [ 42.183970] br0: port 1(vxlan0) entered blocking state [ 42.187274] br0: port 1(vxlan0) entered forwarding state [ 42.258008] veth0: renamed from veth518e4fc [ 42.274049] br0: port 2(veth0) entered blocking state [ 42.278369] br0: port 2(veth0) entered disabled state [ 42.301718] device veth0 entered promiscuous mode [ 42.322007] br0: port 2(veth0) entered blocking state [ 42.325441] br0: port 2(veth0) entered forwarding state [ 42.381440] veth0: renamed from vethb441fa7 [ 42.390995] br0: port 2(veth0) entered blocking state [ 42.394703] br0: port 2(veth0) entered disabled state [ 42.419949] device veth0 entered promiscuous mode [ 42.455566] br0: port 2(veth0) entered blocking state [ 42.459146] br0: port 2(veth0) entered forwarding state [ 42.553731] eth0: renamed from veth38f4811 [ 42.557275] br0: port 2(veth0) entered disabled state [ 42.578364] eth0: renamed from vethf00a2c1 [ 42.581351] br0: port 2(veth0) entered blocking state [ 42.585279] br0: port 2(veth0) entered forwarding state [ 42.753503] veth1: renamed from veth53a6a11 [ 42.760109] br0: port 3(veth1) entered blocking state [ 42.764140] br0: port 3(veth1) entered disabled state [ 42.778016] device veth1 entered promiscuous mode [ 42.787267] br0: port 3(veth1) entered blocking state [ 42.792486] br0: port 3(veth1) entered forwarding state [ 42.851343] veth1: renamed from veth2e663f5 [ 42.858976] br0: port 3(veth1) entered blocking state [ 42.862769] br0: port 3(veth1) entered disabled state [ 42.873106] device veth1 entered promiscuous mode [ 42.879099] br0: port 3(veth1) entered blocking state [ 42.886637] br0: port 3(veth1) entered forwarding state [ 42.919528] veth2: renamed from veth2e0ad97 [ 42.951782] br0: port 4(veth2) entered blocking state [ 42.955681] br0: port 4(veth2) entered disabled state [ 42.969222] device veth2 entered promiscuous mode [ 43.026621] br0: port 4(veth2) entered blocking state [ 43.030765] br0: port 4(veth2) entered forwarding state [ 43.052357] docker_gwbridge: port 2(veth97994ff) entered blocking state [ 43.056607] docker_gwbridge: port 2(veth97994ff) entered disabled state [ 43.070184] device veth97994ff entered promiscuous mode [ 43.099902] br0: port 3(veth1) entered disabled state [ 43.105912] br0: port 3(veth1) entered disabled state [ 43.123911] br0: port 4(veth2) entered disabled state [ 43.132867] IPv6: ADDRCONF(NETDEV_UP): veth97994ff: link is not ready [ 43.349185] veth1: renamed from veth533a7aa [ 43.367877] br0: port 3(veth1) entered blocking state [ 43.372752] br0: port 3(veth1) entered disabled state [ 43.385195] device veth1 entered promiscuous mode [ 43.402888] br0: port 3(veth1) entered blocking state [ 43.406180] br0: port 3(veth1) entered forwarding state [ 43.443450] veth2: renamed from veth57d06ae [ 43.452959] br0: port 4(veth2) entered blocking state [ 43.456440] br0: port 4(veth2) entered disabled state [ 43.484927] device veth2 entered promiscuous mode [ 43.493702] docker_gwbridge: port 3(vethf6c8774) entered blocking state [ 43.498602] docker_gwbridge: port 3(vethf6c8774) entered disabled state [ 43.516657] device vethf6c8774 entered promiscuous mode [ 43.529905] br0: port 4(veth2) entered blocking state [ 43.533439] br0: port 4(veth2) entered forwarding state [ 43.552910] IPv6: ADDRCONF(NETDEV_UP): vethf6c8774: link is not ready [ 43.558841] docker_gwbridge: port 3(vethf6c8774) entered blocking state [ 43.563505] docker_gwbridge: port 3(vethf6c8774) entered forwarding state [ 43.811063] eth0: renamed from vethc4601a7 [ 43.826995] br0: port 3(veth1) entered disabled state [ 43.836057] br0: port 4(veth2) entered disabled state [ 43.840141] docker_gwbridge: port 3(vethf6c8774) entered disabled state [ 43.854887] br0: port 3(veth1) entered blocking state [ 43.858658] br0: port 3(veth1) entered forwarding state [ 44.307722] eth0: renamed from veth470b051 [ 44.319393] br0: port 3(veth1) entered blocking state [ 44.322889] br0: port 3(veth1) entered forwarding state [ 44.604887] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP) [ 44.625890] IPVS: Connection hash table configured (size=4096, memory=64Kbytes) [ 44.636605] IPVS: ipvs loaded. [ 44.816096] eth0: renamed from veth5f6d95f [ 44.826192] br0: port 3(veth1) entered blocking state [ 44.831842] br0: port 3(veth1) entered forwarding state [ 44.861050] eth1: renamed from veth80c9275 [ 44.869304] br0: port 4(veth2) entered blocking state [ 44.873258] br0: port 4(veth2) entered forwarding state [ 44.915722] eth2: renamed from vethc029b67 [ 44.918563] IPv6: ADDRCONF(NETDEV_CHANGE): vethf6c8774: link becomes ready [ 44.929778] docker_gwbridge: port 3(vethf6c8774) entered blocking state [ 44.933795] docker_gwbridge: port 3(vethf6c8774) entered forwarding state [ 44.991744] eth1: renamed from vethf5e53d0 [ 45.003446] br0: port 4(veth2) entered blocking state [ 45.006588] br0: port 4(veth2) entered forwarding state [ 45.036757] eth2: renamed from vethf31d8f0 [ 45.045580] IPv6: ADDRCONF(NETDEV_CHANGE): veth97994ff: link becomes ready [ 45.052366] docker_gwbridge: port 2(veth97994ff) entered blocking state [ 45.056065] docker_gwbridge: port 2(veth97994ff) entered forwarding state [ 45.104525] IPVS: [rr] scheduler registered. [641371.331352] systemd-cgroups invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 [641371.340376] systemd-cgroups cpuset=/ mems_allowed=0 [641371.342280] CPU: 0 PID: 19787 Comm: systemd-cgroups Not tainted 4.19.78-coreos #1 [641371.344578] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017 [641371.347030] Call Trace: [641371.348819] dump_stack+0x5c/0x80 [641371.350525] dump_header+0x6e/0x286 [641371.352117] ? do_try_to_free_pages+0x2ec/0x370 [641371.354210] oom_kill_process.cold.30+0xb/0x1cf [641371.356335] ? oom_badness+0x23/0x140 [641371.358158] out_of_memory+0x1ba/0x470 [641371.359832] __alloc_pages_slowpath+0xbd4/0xcb0 [641371.363256] __alloc_pages_nodemask+0x294/0x2b0 [641371.365922] filemap_fault+0x398/0x610 [641371.368248] ? __wake_up_common_lock+0x89/0xc0 [641371.387568] ? filemap_map_pages+0x1e8/0x350 [641371.391063] ext4_filemap_fault+0x2c/0xab0 [ext4] [641371.395100] __do_fault+0x36/0xf0 [641371.398174] __handle_mm_fault+0xe25/0x11d0 [641371.401598] handle_mm_fault+0xd6/0x200 [641371.405524] __do_page_fault+0x233/0x4c0 [641371.409015] ? async_page_fault+0x8/0x30 [641371.412853] async_page_fault+0x1e/0x30 [641371.417158] RIP: 0033:0x7f05bf39f474 [641371.420008] Code: Bad RIP value. [641371.422174] RSP: 002b:00007fff3cac0f18 EFLAGS: 00010206 [641371.424780] RAX: 00007f05bf39f474 RBX: 00007f05bf850000 RCX: 0000000000000000 [641371.429916] RDX: 00007f05bf5947d8 RSI: 0000000000000000 RDI: 00007f05bf3b4148 [641371.435258] RBP: 00007fff3cac1060 R08: 00007f05bf3b4000 R09: 0000000000000001 [641371.440328] R10: 00007fff3cac0e66 R11: 00007fff3cac0f38 R12: 0000000000000000 [641371.444563] R13: 00007fff3cac0f40 R14: 000000000000001d R15: 00007f05bf8253e0 [641371.450424] Mem-Info: [641371.452030] active_anon:99727 inactive_anon:38392 isolated_anon:0 active_file:164 inactive_file:327 isolated_file:27 unevictable:0 dirty:0 writeback:0 unstable:0 slab_reclaimable:14549 slab_unreclaimable:75984 mapped:22142 shmem:56899 pagetables:1667 bounce:0 free:12166 free_pcp:62 free_cma:0 [641371.471312] Node 0 active_anon:398908kB inactive_anon:153568kB active_file:656kB inactive_file:1308kB unevictable:0kB isolated(anon):0kB isolated(file):108kB mapped:88568kB dirty:0kB writeback:0kB shmem:227596kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 159744kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no [641371.504210] Node 0 DMA free:4448kB min:740kB low:924kB high:1108kB active_anon:6772kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [641371.516810] lowmem_reserve[]: 0 927 927 927 [641371.519205] Node 0 DMA32 free:44216kB min:44312kB low:55388kB high:66464kB active_anon:392136kB inactive_anon:153568kB active_file:656kB inactive_file:1308kB unevictable:0kB writepending:0kB present:1032172kB managed:994072kB mlocked:0kB kernel_stack:3440kB pagetables:6664kB bounce:0kB free_pcp:248kB local_pcp:248kB free_cma:0kB [641371.535415] lowmem_reserve[]: 0 0 0 0 [641371.540051] Node 0 DMA: 34*4kB (UME) 23*8kB (ME) 12*16kB (UM) 5*32kB (ME) 9*64kB (ME) 5*128kB (ME) 2*256kB (M) 4*512kB (ME) 0*1024kB 0*2048kB 0*4096kB = 4448kB [641371.549107] Node 0 DMA32: 188*4kB (UMEH) 53*8kB (UE) 204*16kB (UMEH) 109*32kB (UMEH) 39*64kB (UMEH) 22*128kB (MEH) 11*256kB (ME) 23*512kB (ME) 10*1024kB (ME) 3*2048kB (UM) 0*4096kB = 44216kB [641371.556585] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [641371.561575] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [641371.567074] 57415 total pagecache pages [641371.571566] 0 pages in swap cache [641371.574207] Swap cache stats: add 0, delete 0, find 0/0 [641371.577105] Free swap = 0kB [641371.579251] Total swap = 0kB [641371.581273] 262041 pages RAM [641371.583510] 0 pages HighMem/MovableOnly [641371.586097] 9546 pages reserved [641371.588821] 0 pages hwpoisoned [641371.591759] Tasks state (memory values in pages): [641371.595938] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [641371.604823] [ 510] 0 510 115368 3925 958464 0 0 systemd-journal [641371.611820] [ 543] 0 543 5534 325 81920 0 -1000 systemd-udevd [641371.619417] [ 546] 244 546 3272 202 73728 0 0 systemd-network [641371.627312] [ 600] 245 600 3336 250 73728 0 0 systemd-resolve [641371.636212] [ 601] 997 601 21560 173 73728 0 0 systemd-timesyn [641371.641827] [ 629] 0 629 120482 1616 139264 0 0 authz-broker [641371.647695] [ 641] 0 641 3347 312 61440 0 0 systemd-logind [641371.653817] [ 642] 201 642 4122 383 57344 0 -900 dbus-daemon [641371.663041] [ 643] 0 643 166082 10011 241664 0 -999 containerd [641371.676240] [ 644] 0 644 43284 705 102400 0 0 update_engine [641371.683325] [ 678] 0 678 120891 972 131072 0 0 locksmithd [641371.691112] [ 698] 0 698 1156 35 49152 0 0 agetty [641371.696774] [ 699] 0 699 1156 36 45056 0 0 agetty [641371.705766] [ 720] 0 720 190506 24769 385024 0 -500 dockerd [641371.711148] [ 1324] 0 1324 1877 1151 73728 0 -999 containerd-shim [641371.718018] [ 1339] 999 1339 365876 24353 544768 0 0 mysqld [641371.724467] [ 1399] 0 1399 1893 1082 73728 0 -999 containerd-shim [641371.732127] [ 1424] 0 1424 40861 3471 208896 0 0 traefik [641371.738440] [ 1435] 0 1435 1877 1125 69632 0 -999 containerd-shim [641371.743684] [ 1460] 0 1460 59605 2722 217088 0 0 apache2 [641371.748171] [ 1924] 33 1924 79241 8951 274432 0 0 apache2 [641371.752652] [ 1926] 33 1926 79230 8221 274432 0 0 apache2 [641371.759197] [ 1927] 33 1927 79229 8650 274432 0 0 apache2 [641371.768629] [ 1928] 33 1928 78704 8078 274432 0 0 apache2 [641371.777474] [ 2368] 33 2368 79216 8591 274432 0 0 apache2 [641371.785751] [ 2373] 33 2373 79251 8656 274432 0 0 apache2 [641371.793755] [ 14436] 33 14436 78931 8167 270336 0 0 apache2 [641371.808078] [ 11125] 33 11125 79142 8102 270336 0 0 apache2 [641371.813296] [ 11126] 33 11126 59966 7128 253952 0 0 apache2 [641371.819783] [ 11127] 33 11127 78455 7257 266240 0 0 apache2 [641371.827450] [ 18451] 500 18451 3544 296 69632 0 0 systemd [641371.832928] [ 18458] 500 18458 10522 730 86016 0 0 (sd-pam) [641371.839722] [ 19637] 0 19637 3052 136 61440 0 0 systemd-cgroups [641371.845323] [ 19638] 0 19638 3052 136 61440 0 0 systemd-cgroups [641371.849739] [ 19644] 0 19644 3052 136 65536 0 0 systemd-cgroups [641371.856621] [ 19645] 0 19645 3052 135 61440 0 0 systemd-cgroups [641371.863052] [ 19652] 0 19652 3052 137 61440 0 0 systemd-cgroups [641371.869110] [ 19653] 0 19653 3052 136 61440 0 0 systemd-cgroups [641371.874091] [ 19654] 0 19654 3052 137 61440 0 0 systemd-cgroups [641371.878875] [ 19683] 0 19683 3052 137 65536 0 0 systemd-cgroups [641371.886878] [ 19691] 0 19691 3052 136 61440 0 0 systemd-cgroups [641371.893400] [ 19696] 0 19696 3052 136 65536 0 0 systemd-cgroups [641371.899806] [ 19697] 0 19697 3052 135 61440 0 0 systemd-cgroups [641371.906792] [ 19698] 0 19698 3027 129 57344 0 0 systemd-cgroups [641371.912831] [ 19731] 0 19731 3052 136 61440 0 0 systemd-cgroups [641371.921273] [ 19748] 0 19748 3052 137 61440 0 0 systemd-cgroups [641371.927329] [ 19749] 0 19749 3052 136 61440 0 0 systemd-cgroups [641371.932772] [ 19750] 0 19750 3052 137 61440 0 0 systemd-cgroups [641371.938293] [ 19759] 0 19759 3052 135 65536 0 0 systemd-cgroups [641371.942697] [ 19780] 0 19780 3052 136 61440 0 0 systemd-cgroups [641371.947002] [ 19787] 0 19787 3052 137 65536 0 0 systemd-cgroups [641371.951612] [ 19850] 0 19850 1915 138 53248 0 0 sshd [641371.958532] [ 19876] 0 19876 1857 100 45056 0 0 sshd [641371.964510] [ 19887] 0 19887 3052 136 65536 0 0 systemd-cgroups [641371.970200] [ 19902] 0 19902 3052 135 65536 0 0 systemd-cgroups [641371.976412] [ 19903] 0 19903 3052 135 65536 0 0 systemd-cgroups [641371.981944] [ 19952] 0 19952 1857 63 53248 0 0 sshd [641371.989203] [ 20000] 0 20000 1857 64 53248 0 0 sshd [641371.994926] [ 20013] 0 20013 3027 80 53248 0 0 systemd-cgroups [641372.000221] [ 20058] 0 20058 1857 47 53248 0 0 sshd [641372.006377] [ 20091] 0 20091 1855 42 49152 0 0 sshd [641372.012483] [ 20103] 0 20103 2567 993 49152 0 -500 docker-runc [641372.022187] Out of memory: Kill process 1339 (mysqld) score 96 or sacrifice child [641372.028193] Killed process 1339 (mysqld) total-vm:1463504kB, anon-rss:97412kB, file-rss:0kB, shmem-rss:0kB [641372.037535] oom_reaper: reaped process 1339 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [641374.306339] br0: port 3(veth1) entered disabled state [641374.313350] vethc4601a7: renamed from eth0 [641374.780573] br0: port 3(veth1) entered disabled state [641374.803014] device veth1 left promiscuous mode [641374.806182] br0: port 3(veth1) entered disabled state [641380.979375] veth3: renamed from veth3b3ccb1 [641380.991116] br0: port 3(veth3) entered blocking state [641380.996282] br0: port 3(veth3) entered disabled state [641381.010404] device veth3 entered promiscuous mode [641381.891574] eth0: renamed from veth4a6c895 [641381.910842] br0: port 3(veth3) entered blocking state [641381.914001] br0: port 3(veth3) entered forwarding state
Thanks for the report. Is there a previous Container Linux version where you weren't seeing this problem?
Is there a previous Container Linux version where you weren't seeing this problem?
I can't remember any. IIRC I've seen this problem since I initially set up the VM since July/August.
We have the same issue with some Container Linux VMs on VMware. The following data is from a test server running nothing except one monitoring container.
# uptime
16:26:30 up 7 days, 5:50, 1 user, load average: 0.00, 0.00, 0.00
# cat /proc/meminfo
MemTotal: 2017520 kB
MemFree: 213812 kB
MemAvailable: 315436 kB
Buffers: 6128 kB
Cached: 306624 kB
SwapCached: 0 kB
Active: 289840 kB
Inactive: 179560 kB
Active(anon): 215396 kB
Inactive(anon): 145472 kB
Active(file): 74444 kB
Inactive(file): 34088 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 176 kB
Writeback: 0 kB
AnonPages: 156660 kB
Mapped: 176820 kB
Shmem: 204636 kB
Slab: 756624 kB
SReclaimable: 178672 kB
SUnreclaim: 577952 kB
KernelStack: 4748 kB
PageTables: 2240 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1008760 kB
Committed_AS: 804088 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 557056 kB
HardwareCorrupted: 0 kB
AnonHugePages: 63488 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1124224 kB
DirectMap2M: 972800 kB
DirectMap1G: 0 kB
# slabtop --sort c -o
Active / Total Objects (% used) : 4220674 / 4234894 (99.7%)
Active / Total Slabs (% used) : 59399 / 59399 (100.0%)
Active / Total Caches (% used) : 101 / 127 (79.5%)
Active / Total Size (% used) : 747876.32K / 752120.89K (99.4%)
Minimum / Average / Maximum Object : 0.01K / 0.18K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
538734 538398 99% 0.19K 12827 42 102616K cred_jar
132138 131722 99% 0.59K 2447 54 78304K inode_cache
153856 153815 99% 0.50K 2404 64 76928K kmalloc-512
97704 97659 99% 0.69K 2124 46 67968K sock_inode_cache
67936 67936 100% 1.00K 2123 32 67936K UNIX
63150 63150 100% 1.06K 2105 30 67360K mm_struct
139136 138753 99% 0.25K 2174 64 34784K filp
135936 135680 99% 0.25K 2124 64 33984K skbuff_head_cache
15008 14917 99% 2.00K 938 16 30016K kmalloc-2048
411840 411338 99% 0.06K 6435 64 25740K anon_vma_chain
114492 111382 97% 0.19K 2726 42 21808K dentry
47586 47586 100% 0.38K 1133 42 18128K kmem_cache
198858 198858 100% 0.09K 4323 46 17292K anon_vma
136128 136128 100% 0.12K 2127 64 17016K pid
124740 124740 100% 0.13K 2079 60 16632K kernfs_node_cache
450048 450048 100% 0.03K 3516 128 14064K kmalloc-32
2488 2488 100% 4.00K 311 8 9952K kmalloc-4096
1086976 1086976 100% 0.01K 2123 512 8492K kmalloc-8
5984 5984 100% 1.00K 187 32 5984K kmalloc-1024
55524 55484 99% 0.09K 1322 42 5288K kmalloc-96
19194 17341 90% 0.19K 457 42 3656K kmalloc-192
6384 5493 86% 0.57K 114 56 3648K radix_tree_node
14336 14336 100% 0.25K 224 64 3584K kmalloc-256
384 363 94% 7.50K 96 4 3072K task_struct
47616 47616 100% 0.06K 744 64 2976K kmem_cache_node
2610 1924 73% 1.05K 87 30 2784K ext4_inode_cache
2880 2391 83% 0.66K 60 48 1920K proc_inode_cache
14016 14016 100% 0.12K 219 64 1752K kmalloc-128
24064 23619 98% 0.06K 376 64 1504K kmalloc-64
14547 14161 97% 0.10K 373 39 1492K buffer_head
1978 1426 72% 0.69K 43 46 1376K shmem_inode_cache
27948 24742 88% 0.04K 274 102 1096K numa_policy
405 405 100% 2.06K 27 15 864K sighand_cache
660 660 100% 1.06K 22 30 704K signal_cache
32256 32256 100% 0.02K 126 256 504K kmalloc-16
60 42 70% 8.00K 15 4 480K kmalloc-8192
6384 6384 100% 0.07K 114 56 456K Acpi-Operand
19210 19210 100% 0.02K 113 170 452K avtab_node
56 42 75% 8.00K 14 4 448K biovec-max
598 598 100% 0.69K 13 46 416K files_cache
528 528 100% 0.66K 11 48 352K ovl_inode
306 306 100% 0.94K 9 34 288K RAW
672 672 100% 0.38K 16 42 256K mnt_cache
3520 3520 100% 0.06K 55 64 220K ebitmap_node
1440 1440 100% 0.13K 24 60 192K dm_bufio_buffer-4
140 140 100% 1.12K 5 28 160K RAWv6
3145 3145 100% 0.05K 37 85 148K ftrace_event_field
714 714 100% 0.19K 17 42 136K proc_dir_entry
60 60 100% 2.12K 4 15 128K TCP
100 100 100% 1.25K 4 25 128K UDPv6
1288 1288 100% 0.09K 28 46 112K trace_event_file
320 320 100% 0.25K 5 64 80K pool_workqueue
640 640 100% 0.12K 10 64 80K scsi_sense_cache
84 84 100% 0.75K 2 42 64K task_group
16 16 100% 4.00K 2 8 64K names_cache
78 47 60% 0.81K 2 39 64K bdev_cache
28 28 100% 2.25K 2 14 64K TCPv6
56 56 100% 1.12K 2 28 64K PINGv6
92 92 100% 0.69K 2 46 64K bio-2
840 840 100% 0.07K 15 56 60K eventpoll_pwq
392 392 100% 0.14K 7 56 56K ext4_groupinfo_4k
1224 1224 100% 0.04K 12 102 48K pde_opener
1224 816 66% 0.04K 12 102 48K ext4_extent_status
64 64 100% 0.50K 1 64 32K dma-kmalloc-512
64 64 100% 0.50K 1 64 32K skbuff_fclone_cache
30 30 100% 1.06K 1 30 32K dmaengine-unmap-128
15 15 100% 2.06K 1 15 32K dmaengine-unmap-256
32 32 100% 1.00K 1 32 32K biovec-64
16 16 100% 2.00K 1 16 32K biovec-128
104 104 100% 0.30K 2 52 32K blkdev_requests
13 13 100% 2.40K 1 13 32K request_queue
408 408 100% 0.08K 8 51 32K inotify_inode_mark
52 52 100% 0.61K 1 52 32K hugetlbfs_inode_cache
51 51 100% 0.62K 1 51 32K dio
34 34 100% 0.94K 1 34 32K mqueue_inode_cache
8 8 100% 4.00K 1 8 32K sgpool-128
272 272 100% 0.12K 4 68 32K jbd2_journal_head
44 44 100% 0.72K 1 44 32K fat_inode_cache
102 102 100% 0.31K 2 51 32K nf_conntrack
37 37 100% 0.43K 1 37 16K uts_namespace
1024 1024 100% 0.02K 4 256 16K selinux_file_security
49 49 100% 0.32K 1 49 16K taskstats
64 64 100% 0.25K 1 64 16K dquot
53 53 100% 0.30K 1 53 16K request_sock_TCP
68 68 100% 0.23K 1 68 16K tw_sock_TCP
102 102 100% 0.08K 2 51 8K Acpi-State
146 146 100% 0.05K 2 73 8K Acpi-Parse
51 51 100% 0.16K 1 51 8K sigqueue
40 40 100% 0.20K 1 40 8K file_lock_cache
42 42 100% 0.19K 1 42 8K dmaengine-unmap-16
78 78 100% 0.10K 2 39 8K blkdev_ioc
64 64 100% 0.12K 1 64 8K secpath_cache
39 39 100% 0.20K 1 39 8K ip4-frags
146 146 100% 0.05K 2 73 8K mbcache
64 64 100% 0.12K 1 64 8K ext4_allocation_context
128 128 100% 0.03K 1 128 4K avc_xperms_data
85 85 100% 0.05K 1 85 4K fscrypt_ctx
128 128 100% 0.03K 1 128 4K fscrypt_info
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
64 64 100% 0.06K 1 64 4K ext4_io_end
102 102 100% 0.04K 1 102 4K fat_cache
0 0 0% 0.09K 0 42 0K dma-kmalloc-96
0 0 0% 0.19K 0 42 0K dma-kmalloc-192
0 0 0% 0.01K 0 512 0K dma-kmalloc-8
0 0 0% 0.02K 0 256 0K dma-kmalloc-16
0 0 0% 0.03K 0 128 0K dma-kmalloc-32
0 0 0% 0.06K 0 64 0K dma-kmalloc-64
0 0 0% 0.12K 0 64 0K dma-kmalloc-128
0 0 0% 0.25K 0 64 0K dma-kmalloc-256
0 0 0% 1.00K 0 32 0K dma-kmalloc-1024
0 0 0% 2.00K 0 16 0K dma-kmalloc-2048
0 0 0% 4.00K 0 8 0K dma-kmalloc-4096
0 0 0% 8.00K 0 4 0K dma-kmalloc-8192
0 0 0% 0.12K 0 68 0K iint_cache
0 0 0% 5.50K 0 5 0K net_namespace
0 0 0% 0.45K 0 70 0K user_namespace
0 0 0% 0.94K 0 34 0K PING
0 0 0% 0.31K 0 51 0K xfrm_dst_cache
0 0 0% 0.23K 0 68 0K posix_timers_cache
0 0 0% 0.03K 0 128 0K dnotify_struct
0 0 0% 0.12K 0 68 0K cfq_io_cq
0 0 0% 0.44K 0 36 0K bfq_queue
0 0 0% 0.30K 0 53 0K request_sock_TCPv6
0 0 0% 0.23K 0 68 0K tw_sock_TCPv6
0 0 0% 2.57K 0 12 0K dm_uevent
0 0 0% 3.23K 0 9 0K kcopyd_job
0 0 0% 0.21K 0 37 0K nf_conntrack_expect
Thanks for the report. Is there a previous Container Linux version where you weren't seeing this problem?
@bgilbert: The problems started when we upgraded from 2135.6.0 to 2191.4.1.
This problem seems still to exist on stable(2345.3.0). As a matter of fact it's the first version I experienced it although the machine has been running without any configuration changes or significant changes in load since end of November.
cred_jar is the top slab consumer. Disregarding inode caches, which should be mostly reclaimable I guess, UNIX is very high. Wasn't immediately able to find UNIX slabs in the kernel source, my guess would be they are related to AF_UNIX sockets.
I have another instance running, which has much higher load (number of processes) but it does not suffer from the problem. The only configuration difference between the affected and non-affected instance is that the affected one listens for ssh on port 443, the non-affected one has port 443 firewalled.
If I firewall port 443 on the affected machine the cred_jar consumption stops growing immediately, but it never goes down again.
This looks like a kernel bug to me. Or can systemd do something wrong in the socket passing mechanism for every connection attempt on 443? There are of course many attempts an hour, but nothing dramatic the system should not be able to handle. ss does not show any abnormal socket usage, neither for tcp nor unix.
With the remaining lifetime of CoreOS I am not holding my breath to get this fixed. But I don't think this is really a distro bug, it's probably upstream somewhere. If somebody can provide a link to any upstream bug/fix that would be useful information.
Same issue here, my release is:
# cat /etc/lsb-release
DISTRIB_ID="Container Linux by CoreOS"
DISTRIB_RELEASE=2303.3.0
DISTRIB_CODENAME="Rhyolite"
DISTRIB_DESCRIPTION="Container Linux by CoreOS 2303.3.0 (Rhyolite)"
and my slabinfo:
# slabtop --sort c -o
Active / Total Objects (% used) : 739390584 / 740008326 (99.9%)
Active / Total Slabs (% used) : 11594275 / 11594275 (100.0%)
Active / Total Caches (% used) : 105 / 129 (81.4%)
Active / Total Size (% used) : 47121380.33K / 47376581.93K (99.5%)
Minimum / Average / Maximum Object : 0.01K / 0.06K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
734506368 734506368 100% 0.06K 11476662 64 45906648K ebitmap_node
328160 80830 24% 0.50K 10255 32 164080K kmalloc-512
69442 36292 52% 2.00K 4341 16 138912K kmalloc-2048
13148 12571 95% 7.50K 3287 4 105184K task_struct
85359 75134 88% 1.05K 2857 30 91424K ext4_inode_cache
462336 459563 99% 0.19K 11008 42 88064K cred_jar
382641 323093 84% 0.19K 9125 42 73000K dentry
251968 249625 99% 0.25K 7874 32 62992K filp
51488 41972 81% 1.00K 1609 32 51488K kmalloc-1024
78846 77530 98% 0.59K 1461 54 46752K inode_cache
69342 68580 98% 0.66K 1449 48 46368K proc_inode_cache
70000 63132 90% 0.57K 2500 28 40000K radix_tree_node
56902 56673 99% 0.69K 1237 46 39584K sock_inode_cache
57504 54982 95% 0.66K 1198 48 38336K ovl_inode
9056 9007 99% 4.00K 1132 8 36224K kmalloc-4096
31756 31756 100% 1.00K 995 32 31840K UNIX
504192 501166 99% 0.06K 7878 64 31512K anon_vma_chain
27660 27660 100% 1.06K 922 30 29504K signal_cache
336950 335179 99% 0.09K 7325 46 29300K anon_vma
26400 26400 100% 1.06K 880 30 28160K mm_struct
38042 38042 100% 0.69K 827 46 26464K files_cache
12315 12298 99% 2.06K 821 15 26272K sighand_cache
3001 3001 100% 8.00K 763 4 24416K kmalloc-8192
74336 73888 99% 0.25K 2323 32 18584K skbuff_head_cache
186102 63644 34% 0.09K 4431 42 17724K kmalloc-96
24104 22322 92% 0.69K 524 46 16768K shmem_inode_cache
527360 479425 90% 0.03K 4120 128 16480K kmalloc-32
140439 137220 97% 0.10K 3601 39 14404K buffer_head
10075 10075 100% 1.25K 403 25 12896K UDPv6
183808 158004 85% 0.06K 2872 64 11488K kmalloc-64
60102 47918 79% 0.19K 1431 42 11448K kmalloc-192
84704 84704 100% 0.12K 2647 32 10588K pid
72450 72243 99% 0.13K 2415 30 9660K kernfs_node_cache
131152 131152 100% 0.07K 2342 56 9368K Acpi-Operand
4020 4020 100% 2.12K 268 15 8576K TCP
6936 6936 100% 0.94K 204 34 6528K RAW
118320 107640 90% 0.04K 1160 102 4640K numa_policy
11340 11191 98% 0.38K 270 42 4320K mnt_cache
1750 1750 100% 2.25K 125 14 4000K TCPv6
3472 3360 96% 1.12K 124 28 3968K RAWv6
14976 14893 99% 0.25K 468 32 3744K kmalloc-256
29728 25895 87% 0.12K 929 32 3716K kmalloc-128
86190 86190 100% 0.04K 845 102 3380K pde_opener
44240 44240 100% 0.07K 790 56 3160K eventpoll_pwq
7392 7222 97% 0.38K 176 42 2816K kmem_cache
329728 329728 100% 0.01K 644 512 2576K kmalloc-8
320 300 93% 8.00K 80 4 2560K biovec-max
142080 136981 96% 0.02K 555 256 2220K kmalloc-16
13248 13248 100% 0.12K 414 32 1656K secpath_cache
6432 5952 92% 0.25K 201 32 1608K pool_workqueue
10540 10540 100% 0.12K 310 34 1240K jbd2_journal_head
2400 2400 100% 0.50K 75 32 1200K skbuff_fclone_cache
29886 29886 100% 0.04K 293 102 1172K ext4_extent_status
3672 3431 93% 0.31K 72 51 1152K nf_conntrack
17664 17472 98% 0.06K 276 64 1104K ext4_io_end
1518 1518 100% 0.69K 33 46 1056K bio-2
1344 1344 100% 0.75K 32 42 1024K task_group
256 256 100% 4.00K 32 8 1024K names_cache
1024 1024 100% 1.00K 32 32 1024K biovec-64
1632 1632 100% 0.62K 32 51 1024K dio
4352 4352 100% 0.23K 128 34 1024K tw_sock_TCPv6
1120 1120 100% 0.91K 32 35 1024K sw_flow
5292 5292 100% 0.19K 126 42 1008K proc_dir_entry
6748 6748 100% 0.14K 241 28 964K ext4_groupinfo_4k
3536 3536 100% 0.23K 104 34 832K tw_sock_TCP
286 286 100% 2.75K 26 11 832K iommu_domain
816 816 100% 0.94K 24 34 768K mqueue_inode_cache
14016 14016 100% 0.05K 192 73 768K mbcache
819 377 46% 0.81K 21 39 672K bdev_cache
24480 24480 100% 0.02K 144 170 576K avtab_node
2940 2940 100% 0.19K 70 42 560K dmaengine-unmap-16
85 85 100% 5.50K 17 5 544K net_namespace
1568 1568 100% 0.32K 32 49 512K taskstats
1696 1696 100% 0.30K 32 53 512K request_sock_TCP
8128 8128 100% 0.06K 127 64 508K kmem_cache_node
1073 1073 100% 0.43K 29 37 464K uts_namespace
3420 3360 98% 0.13K 114 30 456K dm_bufio_buffer-4
1404 1404 100% 0.30K 27 52 432K blkdev_requests
169 98 57% 2.40K 13 13 416K request_queue
3328 3328 100% 0.12K 104 32 416K scsi_sense_cache
5049 5049 100% 0.08K 99 51 396K inotify_inode_mark
23040 23040 100% 0.02K 90 256 360K selinux_file_security
10240 10240 100% 0.03K 80 128 320K fscrypt_info
1120 1120 100% 0.25K 35 32 280K dquot
1734 1734 100% 0.16K 34 51 272K sigqueue
5525 5525 100% 0.05K 65 85 260K ftrace_event_field
1280 1280 100% 0.20K 32 40 256K file_lock_cache
1088 1088 100% 0.23K 32 34 256K posix_timers_cache
1092 1092 100% 0.20K 28 39 224K ip4-frags
1564 1564 100% 0.09K 34 46 136K trace_event_file
1287 1287 100% 0.10K 33 39 132K blkdev_ioc
2336 2336 100% 0.05K 32 73 128K Acpi-Parse
4096 4096 100% 0.03K 32 128 128K avc_xperms_data
208 157 75% 0.61K 4 52 128K hugetlbfs_inode_cache
2720 2720 100% 0.05K 32 85 128K fscrypt_ctx
1024 1024 100% 0.12K 32 32 128K ext4_allocation_context
88 88 100% 0.72K 2 44 64K fat_inode_cache
30 30 100% 1.06K 1 30 32K dmaengine-unmap-128
15 15 100% 2.06K 1 15 32K dmaengine-unmap-256
16 16 100% 2.00K 1 16 32K biovec-128
1024 1024 100% 0.03K 8 128 32K dnotify_struct
8 8 100% 4.00K 1 8 32K sgpool-128
306 306 100% 0.08K 6 51 24K Acpi-State
32 32 100% 0.50K 1 32 16K dma-kmalloc-512
512 512 100% 0.02K 2 256 8K jbd2_revoke_table_s
0 0 0% 0.09K 0 42 0K dma-kmalloc-96
0 0 0% 0.19K 0 42 0K dma-kmalloc-192
0 0 0% 0.01K 0 512 0K dma-kmalloc-8
0 0 0% 0.02K 0 256 0K dma-kmalloc-16
0 0 0% 0.03K 0 128 0K dma-kmalloc-32
0 0 0% 0.06K 0 64 0K dma-kmalloc-64
0 0 0% 0.12K 0 32 0K dma-kmalloc-128
0 0 0% 0.25K 0 32 0K dma-kmalloc-256
0 0 0% 1.00K 0 32 0K dma-kmalloc-1024
0 0 0% 2.00K 0 16 0K dma-kmalloc-2048
0 0 0% 4.00K 0 8 0K dma-kmalloc-4096
0 0 0% 8.00K 0 4 0K dma-kmalloc-8192
0 0 0% 0.12K 0 34 0K iint_cache
0 0 0% 0.45K 0 35 0K user_namespace
0 0 0% 0.94K 0 34 0K PING
0 0 0% 0.31K 0 51 0K xfrm_dst_cache
0 0 0% 0.12K 0 34 0K cfq_io_cq
0 0 0% 0.30K 0 53 0K request_sock_TCPv6
0 0 0% 1.12K 0 28 0K PINGv6
0 0 0% 2.57K 0 12 0K dm_uevent
0 0 0% 3.23K 0 9 0K kcopyd_job
0 0 0% 0.04K 0 102 0K fat_cache
0 0 0% 0.21K 0 37 0K nf_conntrack_expect
0 0 0% 0.09K 0 42 0K nf_conncount_rb
The ebitmap_node has eaten over 40GB of memory, and after googling, I found out that this part of slabs is allocated by SELinux, and I'm stucked here. Any one has any clue?
