mptcp
mptcp copied to clipboard
Published
20 hours ago •
multipath-tcp
multipath-tcp
Use after free warning in lib/refcount
Hello guys, I've found this error in one of my server. Nothing else seems happening around it but let me know if I can do anything to provide more information.
[16751.877150] ------------[ cut here ]------------
[16751.877151] refcount_t: underflow; use-after-free.
[16751.877173] WARNING: CPU: 1 PID: 17 at lib/refcount.c:187 refcount_sub_and_test_checked+0x3e/0x50
[16751.877174] Modules linked in: cls_fw cls_u32 xt_tcpudp sch_htb xt_policy drbg ansi_cprng authenc echainiv xfrm6_mode_tunnel xfrm4_mode_tunnel ipip fou ip_tunnel ip6_udp_tunnel udp_tunnel tun xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key xfrm_algo dummy ip6table_filter ip6_tables nfit libnvdimm crct10dif_pclmul iptable_filter crc32_pclmul xt_statistic xt_mark ghash_clmulni_intel xt_connmark pcbc xt_conntrack xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_addrtype iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat aesni_intel nf_conntrack aes_x86_64 crypto_simd nf_defrag_ipv6 nf_defrag_ipv4 cryptd glue_helper libcrc32c ppdev snd_pcsp evdev snd_pcm intel_rapl_perf snd_timer serio_raw snd soundcore parport_pc parport button mptcp_fullmesh ip_tables x_tables ext4 crc32c_generic
[16751.877209] crc16 mbcache jbd2 fscrypto dm_mod crc32c_intel nvme ena nvme_core i2c_piix4
[16751.877217] CPU: 1 PID: 17 Comm: ksoftirqd/1 Not tainted 4.19.67-mat-0.95 #8
[16751.877217] Hardware name: Amazon EC2 m5.xlarge/, BIOS 1.0 10/16/2017
[16751.877219] RIP: 0010:refcount_sub_and_test_checked+0x3e/0x50
[16751.877220] Code: 75 0c f0 0f b1 16 75 27 85 d2 0f 94 c0 c3 80 3d 3c 7a d0 00 00 75 15 48 c7 c7 58 b9 87 b8 c6 05 2c 7a d0 00 01 e8 22 ac c8 ff <0f> 0b 31 c0 c3 83 f8 ff 75 bf eb f6 66 0f 1f 44 00 00 48 89 fe bf
[16751.877222] RSP: 0018:ffffb56681933d80 EFLAGS: 00010282
[16751.877223] RAX: 0000000000000000 RBX: ffff9457540a9f80 RCX: 0000000000000006
[16751.877223] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9457d2a966b0
[16751.877224] RBP: ffff94574fe82bc0 R08: 0000000000000001 R09: 00000000000001da
[16751.877225] R10: ffff94574fe82bc0 R11: 0000000000000000 R12: ffff94579c1483f8
[16751.877226] R13: ffffffffb807c490 R14: dead000000000200 R15: 0000000000000003
[16751.877227] FS: 0000000000000000(0000) GS:ffff9457d2a80000(0000) knlGS:0000000000000000
[16751.877228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[16751.877228] CR2: 00007f8de3ca8ff8 CR3: 000000011b60a006 CR4: 00000000007606e0
[16751.877231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[16751.877232] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[16751.877233] PKRU: 55555554
[16751.877233] Call Trace:
[16751.877237] mptcp_mpcb_put+0x12/0x50
[16751.877240] mptcp_sock_destruct+0x71/0xf0
[16751.877243] __sk_destruct+0x24/0x1c0
[16751.877247] call_timer_fn+0x2b/0x130
[16751.877249] run_timer_softirq+0x1d3/0x420
[16751.877251] ? __switch_to_asm+0x41/0x70
[16751.877252] ? __switch_to+0x8c/0x450
[16751.877253] ? __switch_to_asm+0x41/0x70
[16751.877254] ? __switch_to_asm+0x35/0x70
[16751.877257] __do_softirq+0x10d/0x2c3
[16751.877261] run_ksoftirqd+0x26/0x40
[16751.877264] smpboot_thread_fn+0x10e/0x160
[16751.877266] kthread+0xf8/0x130
[16751.877268] ? sort_range+0x20/0x20
[16751.877269] ? kthread_create_worker_on_cpu+0x70/0x70
[16751.877270] ret_from_fork+0x35/0x40
[16751.877272] ---[ end trace 2dc0947227a2bc2a ]---
There is one place in mptcp_sub_close where we inc the refcnt and then schedule the work-queue. That could fail though, so ideally we should decrement again.
However, that does not explain the issue with the underflow.
@cifvts - Are there any other warnings,... above this particular warning here?
I looked all my server and all of them has the same warning happening after a while. Nothing else seems to happens around that time but I've attached the logs so you can look better. logs.tar.gz
Thanks for the logs! It's yet unclear to me how this happens. I'm trying to trigger it with syzkaller. Let's see...
sorry to nercobump but adding this in case it helps
using trunk branch
uname -a
Linux test 5.4.69 #0 SMP Fri Oct 9 01:10:05 2020 aarch64 GNU/Linux
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.717021] ------------[ cut here ]------------
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.721650] refcount_t: increment on 0; use-after-free.
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.726907] WARNING: CPU: 0 PID: 0 at lib/refcount.c:156 refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.734902] Modules linked in: ath9k ath9k_htc ath9k_common rtl8192cu rtl8192c_common rtl_usb rt2800usb rt2800lib qcserial pppoe ppp_async option ipw cdc_mbim brcmfmac ath9k_hw ath usb_wwan usb_serial_simple ti_usb_3410_5052 sr9700 smsc95xx smsc75xx sierra_net sierra rtlwifi rtl8xxxu rtl8187 rt2x00usb rt2x00lib rndis_host qmi_wwan pppox ppp_generic pl2303 oti6858 mt7601u mos7720 mmc_spi mct_u232 mcs7830 mac80211 lzo keyspan kalmia ipt_REJECT huawei_cdc_ncm garmin_gps ftdi_sio ebtable_nat ebtable_filter ebtable_broute dm9601 cypress_m8 cp210x ch341 cfg80211 cdc_subset cdc_ncm cdc_ether cdc_eem belkin_sa ax88179_178a asix ark3116 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_socket xt_recent xt_quota xt_policy xt_pkttype xt_owner xt_ndpi xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iface xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_condition xt_comment xt_addrtype xt_TRACE xt_TPROXY xt_TCPMSS xt_REDIRECT
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.734968] xt_MASQUERADE xt_LOG xt_IPMARK xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY xt_ACCOUNT visor via_velocity via_rhine usbserial usbnet usbhid tulip ts_fsm ts_bm solos_pci slhc sky2 skge sis900 sis190 sch_cake rtl8150 r8712u(C) r8169 r8152 r6040 pegasus pcnet32 of_mmc_spi nf_tproxy_ipv6 nf_tproxy_ipv4 nf_socket_ipv6 nf_socket_ipv4 nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast ts_kmp nf_conntrack_amanda nf_conncount ne2k_pci macvlan lzo_decompress lzo_compress kaweth iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ipheth ip6table_raw ip_tables hso hid_generic forcedeth ezusb ethoc et131x ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.821794] ebt_limit ebt_among ebt_802_3 e1000e e100 crc7 crc_itu_t crc_ccitt compat_xtables compat cdc_wdm cdc_acm brcmutil bnx2 atl2 atl1e atl1c atl1 asn1_decoder arptable_filter arpt_mangle arp_tables 8390 8250_pci 8139too 8139cp tcp_nanqinlang sch_teql sch_sfq sch_red sch_prio sch_pie sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_police act_pedit act_ipt act_gact act_csum sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred snd_bcm2835(C) hid evdev i2c_gpio i2c_algo_bit i2c_dev spi_ks8995 ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97442.908969] nf_defrag_ipv4 ip6t_NPT nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 swconfig 3c59x ip6_gre ip_gre gre vmxnet3 e1000 ifb nat46 sit sctp libcrc32c ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ip6_tunnel netlink_diag tunnel6 tunnel4 ip_tunnel hfcpci hfcmulti veth tun snd_rawmidi snd_seq_device snd_pcm_oss snd_pcm_dmaengine snd_pcm snd_timer snd_mixer_oss snd_hwdep snd_compress snd soundcore mISDN_dsp l1oip mISDN_core xfrm_user xfrm_ipcomp af_key xfrm_algo autofs4 br2684 atm nls_utf8 zram zsmalloc natsemi eeprom_93cx6 sha1_generic md5 ghash_generic gf128mul gcm echainiv des_generic libdes deflate zlib_inflate zlib_deflate authenc crypto_acompress vfat fat nls_iso8859_1 nls_cp437 ahci libahci libata fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd gpio_button_hotplug tg3 b44 ssb ptp realtek pps_core
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.074406] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G C 5.4.69 #0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.081706] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.087531] pstate: 60400005 (nZCv daif +PAN -UAO)
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.092314] pc : refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.096836] lr : refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.101356] sp : ffffffc010003df0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.104660] x29: ffffffc010003df0 x28: ffffffc0109c50c0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.109964] x27: ffffffc0109ad598 x26: ffffff807fb997b0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.115267] x25: 0000000000000002 x24: dead000000000100
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.120570] x23: dead000000000122 x22: ffffff80775cb688
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.125873] x21: ffffffc010ab2000 x20: ffffff8073aab600
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.131175] x19: ffffff80775cb600 x18: 0000000000000000
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.136478] x17: 0000000000000000 x16: 0000000000000001
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.141781] x15: 0000000000000000 x14: 0720072007200720
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.147084] x13: 0720072007200720 x12: 0720072007200720
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.152386] x11: 0720072007200720 x10: 0720072007200720
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.157689] x9 : 07200720072e0765 x8 : 076507720766072d
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.162992] x7 : 0772076507740766 x6 : 0000000000000001
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.168294] x5 : ffffffc0103908b0 x4 : 0000000000000001
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.173597] x3 : ffffffc0109c8ce4 x2 : 0000000000000004
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.178900] x1 : 0000000000000004 x0 : 000000000000002b
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.184203] Call trace:
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.186641] refcount_inc_checked+0x40/0x48
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.190819] tcp_tsq_handler+0x108/0x160
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.194733] tcp_tasklet_func+0xc8/0x100
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.198648] tasklet_action_common.isra.19+0xac/0x150
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.203691] tasklet_action+0x24/0x30
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.207344] __do_softirq+0x11c/0x250
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.210997] irq_exit+0x9c/0xb8
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.214129] __handle_domain_irq+0x64/0xb8
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.218215] gic_handle_irq+0x5c/0xb8
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.221868] el1_irq+0xf0/0x1c0
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.225001] arch_cpu_idle+0x10/0x18
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.228568] do_idle+0x1e4/0x258
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.231787] cpu_startup_entry+0x24/0x78
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.235701] rest_init+0xb0/0xbc
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.238921] arch_call_rest_init+0xc/0x14
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.242921] start_kernel+0x3c4/0x3dc
Fri Nov 27 12:23:48 2020 kern.warn kernel: [97443.246573] ---[ end trace b8b637b795f1dacf ]---
