tempesta icon indicating copy to clipboard operation
tempesta copied to clipboard

Warnings during h2 message exchanges

Open vankoven opened this issue 5 years ago • 6 comments

Caught a few warnings during testing h2 with h2spec:

  • [ ] 1: Can be reproduced with ./h2spec http2/5.1.1/3 -h <host> -p 443 -t -k
[92286.039416] ------------[ cut here ]------------
[92286.041515] WARNING: CPU: 0 PID: 0 at /home/user/qtc/release/tempesta/tempesta_fw/http.c:5722 tfw_http_msg_process_generic+0x7d0/0xd10 [tempesta_fw]
[92286.047585] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92286.061263]  [last unloaded: tempesta_lib]
[92286.061982] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92286.063774] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92286.065379] task: ffffffff83e12480 task.stack: ffffffff83e00000
[92286.066766] RIP: 0010:tfw_http_msg_process_generic+0x7d0/0xd10 [tempesta_fw]
[92286.070566] RSP: 0018:ffff917165403910 EFLAGS: 00010246
[92286.071722] RAX: 0000000000000000 RBX: ffff9170da1821a0 RCX: 0000000000000000
[92286.073611] RDX: ffff917165403a20 RSI: 0000000000000000 RDI: ffff9170da1821a0
[92286.075043] RBP: 0000000000000000 R08: ffff917165403940 R09: ffff9170da182908
[92286.076385] R10: 00000000ffffffff R11: 000000000000001d R12: ffff9170da1828c0
[92286.077589] R13: 0000000000000000 R14: ffff9170da1829c8 R15: 0000000000000000
[92286.078756] FS:  0000000000000000(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92286.080118] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92286.081061] CR2: 00007f5cd152c000 CR3: 000000000520a004 CR4: 00000000003606f0
[92286.082235] Call Trace:
[92286.082667]  <IRQ>
[92286.084194]  ? gcmaes_decrypt.constprop.13+0x11e/0x2a0 [aesni_intel]
[92286.085448]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92286.086582]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92286.087444]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92286.088410]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92286.089340]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92286.090138]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92286.090979]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92286.093377]  ? lock_timer_base+0x74/0x90
[92286.094168]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92286.095463]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92286.097068]  ? kvm_sched_clock_read+0x1a/0x30
[92286.099107]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92286.100439]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92286.102181]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92286.103946]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92286.107233]  tcp_rcv_established+0x4d2/0x570
[92286.109332]  tcp_v4_do_rcv+0x129/0x1d0
[92286.111040]  tcp_v4_rcv+0x947/0xa50
[92286.112838]  ip_local_deliver_finish+0x9a/0x1c0
[92286.114862]  ip_local_deliver+0x6b/0xe0
[92286.116076]  ? tcp_v4_early_demux+0x112/0x150
[92286.117433]  ? ip_rcv_finish+0x17a/0x400
[92286.118383]  ip_rcv+0x289/0x3c0
[92286.119066]  ? inet_del_offload+0x40/0x40
[92286.120930]  __netif_receive_skb_core+0x84f/0xb30
[92286.124403]  ? enqueue_entity+0x110/0x7a0
[92286.126422]  ? process_backlog+0xa3/0x160
[92286.127427]  process_backlog+0xa3/0x160
[92286.129167]  net_rx_action+0x28e/0x3f0
[92286.130859]  __do_softirq+0x10f/0x2a8
[92286.131956]  irq_exit+0xae/0xb0
[92286.132633]  call_function_single_interrupt+0x7d/0x90
[92286.134064]  </IRQ>
[92286.134666] RIP: 0010:native_safe_halt+0x2/0x10
[92286.136918] RSP: 0018:ffffffff83e03e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[92286.139947] RAX: ffffffff8349a7a0 RBX: ffffffff83e12480 RCX: 0000000000000000
[92286.142009] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[92286.144613] RBP: 0000000000000000 R08: 0000000000000002 R09: 000000000000b7bf
[92286.146234] R10: ffffb17ec0663d78 R11: 0000000000000000 R12: ffffffff83e12480
[92286.147420] R13: ffffffff83e12480 R14: 0000000000000000 R15: 0000000000000000
[92286.150573]  ? __sched_text_end+0x3/0x3
[92286.152340]  default_idle+0x1a/0xf0
[92286.153914]  do_idle+0x16e/0x1f0
[92286.155376]  cpu_startup_entry+0x6f/0x80
[92286.157434]  start_kernel+0x467/0x487
[92286.158812]  secondary_startup_64+0xa5/0xb0
[92286.160053] Code: 00 f6 c4 08 0f 85 51 04 00 00 48 8b 44 24 10 48 89 43 40 49 89 c6 e9 de f9 ff ff 48 83 7f 08 00 0f 85 8a f8 ff ff e9 87 f8 ff ff <0f> 0b 41 bf ea ff ff ff e9 73 f9 ff ff 49 89 7c 24 10 48 89 3f 
[92286.165460] ---[ end trace ab4f9128609f842b ]---
  • [x] 2 (#1617):
[92308.274202] ------------[ cut here ]------------
[92308.275475] WARNING: CPU: 1 PID: 16 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:1199 tfw_hpack_decode+0x1c32/0x1cc0 [tempesta_fw]
[92308.278623] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92308.294190]  [last unloaded: tempesta_lib]
[92308.294924] CPU: 1 PID: 16 Comm: ksoftirqd/1 Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92308.296969] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92308.298523] task: ffff9171415d3c00 task.stack: ffffb17ec06b0000
[92308.299771] RIP: 0010:tfw_hpack_decode+0x1c32/0x1cc0 [tempesta_fw]
[92308.300828] RSP: 0018:ffffb17ec06b36c0 EFLAGS: 00010206
[92308.301697] RAX: 0000000000000008 RBX: ffff9170e2511020 RCX: 0000000000000000
[92308.302886] RDX: 0000000000000000 RSI: 00000000000000e0 RDI: ffff9170dd5cf518
[92308.304062] RBP: ffffffffc08bcb00 R08: ffff9170e2eb802a R09: 0000000000000020
[92308.305817] R10: ffffffffc08bcda0 R11: 000000000000000a R12: ffff9170d713e6a8
[92308.307010] R13: 0000000000000000 R14: ffff9170d713e6a8 R15: ffff9170dd5cf4d0
[92308.308187] FS:  0000000000000000(0000) GS:ffff917165500000(0000) knlGS:0000000000000000
[92308.309617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92308.310565] CR2: 00007f5cd1539020 CR3: 000000000520a002 CR4: 00000000003606e0
[92308.311741] Call Trace:
[92308.312176]  ? tfw_h2_add_stream+0x5a/0xf0 [tempesta_fw]
[92308.313124]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.314147]  tfw_h2_parse_req+0x50/0x180 [tempesta_fw]
[92308.315096]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.316598]  ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.317477]  tfw_http_msg_process_generic+0x1dc/0xd10 [tempesta_fw]
[92308.319349]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92308.321086]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.323719]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92308.326659]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92308.327982]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92308.329865]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.330764]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92308.331456]  ? lock_timer_base+0x74/0x90
[92308.332013]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92308.333345]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.335232]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92308.337458]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92308.338769]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92308.341303]  ? __skb_flow_dissect+0x659/0x10c0
[92308.342467]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92308.344613]  tcp_rcv_established+0x4d2/0x570
[92308.346180]  tcp_v4_do_rcv+0x129/0x1d0
[92308.347768]  tcp_v4_rcv+0x947/0xa50
[92308.348579]  ip_local_deliver_finish+0x9a/0x1c0
[92308.349812]  ip_local_deliver+0x6b/0xe0
[92308.350690]  ? tcp_v4_early_demux+0x112/0x150
[92308.352299]  ? ip_rcv_finish+0x17a/0x400
[92308.353356]  ip_rcv+0x289/0x3c0
[92308.354171]  ? inet_del_offload+0x40/0x40
[92308.356063]  __netif_receive_skb_core+0x84f/0xb30
[92308.357912]  ? vring_unmap_one+0x16/0x70 [virtio_ring]
[92308.359478]  ? virtqueue_napi_complete+0x39/0x70 [virtio_net]
[92308.360340]  ? process_backlog+0xa3/0x160
[92308.360930]  process_backlog+0xa3/0x160
[92308.361507]  net_rx_action+0x28e/0x3f0
[92308.362052]  __do_softirq+0x10f/0x2a8
[92308.362679]  run_ksoftirqd+0x1c/0x40
[92308.363754]  smpboot_thread_fn+0x10e/0x160
[92308.364496]  kthread+0xff/0x130
[92308.365048]  ? sort_range+0x20/0x20
[92308.365647]  ? kthread_create_on_node+0x70/0x70
[92308.366405]  ret_from_fork+0x35/0x40
[92308.367003] Code: 89 74 24 48 e9 4c fe ff ff 4d 89 e6 e9 e5 f3 ff ff 45 89 ea 49 89 ee 31 c9 e9 9b f6 ff ff 45 89 ea 31 c9 e9 98 fc ff ff 4d 89 e6 <0f> 0b 41 ba fe ff ff ff e9 75 ef ff ff 49 c1 e5 05 48 8b 74 24 
[92308.370154] ---[ end trace ab4f9128609f842c ]---
  • [ ] 3: Can be reproduced with ./h2spec http2/8.1.2.1/3 -h <host> -p 443 -t -k
[92308.408692] ------------[ cut here ]------------
[92308.410253] WARNING: CPU: 0 PID: 15856 at /home/user/qtc/release/tempesta/tempesta_fw/http_msg.c:528 tfw_http_msg_hdr_close+0x128/0x200 [tempesta_fw]
[92308.414407] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92308.427320]  [last unloaded: tempesta_lib]
[92308.428045] CPU: 0 PID: 15856 Comm: systemd-journal Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92308.430095] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92308.432347] task: ffff917133d1cb00 task.stack: ffffb17ec0a94000
[92308.433969] RIP: 0010:tfw_http_msg_hdr_close+0x128/0x200 [tempesta_fw]
[92308.435336] RSP: 0018:ffff917165403790 EFLAGS: 00010293
[92308.437801] RAX: 0000000000008800 RBX: ffff9170dd4e2680 RCX: 0000000000000000
[92308.440525] RDX: 000000000000000b RSI: 0000000000000040 RDI: ffff9170df80b020
[92308.443259] RBP: ffff9170df80b020 R08: ffff9171654038c4 R09: 0000000000000001
[92308.446107] R10: 00000000ffffffff R11: 000000000000000a R12: ffff9170d713c099
[92308.447880] R13: 0000000000000000 R14: ffff9170df80b2c0 R15: ffff9170df80b2c8
[92308.449896] FS:  00007f5cd3e7c8c0(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92308.451945] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92308.453580] CR2: 00007f5cd153b000 CR3: 0000000081152001 CR4: 00000000003606f0
[92308.455056] Call Trace:
[92308.455612]  <IRQ>
[92308.456009]  tfw_hpack_decode+0xc2d/0x1cc0 [tempesta_fw]
[92308.456954]  ? gcmaes_decrypt.constprop.13+0x73/0x2a0 [aesni_intel]
[92308.458501]  ? __save_stack_trace+0x82/0x100
[92308.459339]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.460384]  tfw_h2_parse_req+0x50/0x180 [tempesta_fw]
[92308.461277]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92308.462293]  ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.463143]  tfw_http_msg_process_generic+0x1dc/0xd10 [tempesta_fw]
[92308.465235]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92308.466305]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.467172]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92308.468128]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92308.469067]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92308.469874]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.470721]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92308.471621]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92308.472561]  ? ip_output+0x71/0xe0
[92308.473174]  ? ip_queue_xmit+0x5c/0x3b0
[92308.473835]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.474661]  ? kvm_sched_clock_read+0x1a/0x30
[92308.475460]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92308.476384]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92308.477364]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92308.478232]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92308.479632]  tcp_rcv_established+0x4d2/0x570
[92308.480501]  tcp_v4_do_rcv+0x129/0x1d0
[92308.481246]  tcp_v4_rcv+0x947/0xa50
[92308.481906]  ip_local_deliver_finish+0x9a/0x1c0
[92308.482771]  ip_local_deliver+0x6b/0xe0
[92308.483341]  ? tcp_v4_early_demux+0x112/0x150
[92308.484009]  ? ip_rcv_finish+0x17a/0x400
[92308.484613]  ip_rcv+0x289/0x3c0
[92308.485173]  ? inet_del_offload+0x40/0x40
[92308.486567]  __netif_receive_skb_core+0x84f/0xb30
[92308.488025]  ? tcp_write_xmit+0x2a7/0x11c0
[92308.489469]  ? tfw_wq_pop_ticket+0x7e/0xf0 [tempesta_fw]
[92308.491686]  ? process_backlog+0xa3/0x160
[92308.492666]  process_backlog+0xa3/0x160
[92308.493544]  net_rx_action+0x28e/0x3f0
[92308.494455]  __do_softirq+0x10f/0x2a8
[92308.495968]  irq_exit+0xae/0xb0
[92308.497052]  call_function_single_interrupt+0x7d/0x90
[92308.498692]  </IRQ>
[92308.500423] RIP: 0010:___bpf_prog_run+0x851/0x1270
[92308.501774] RSP: 0018:ffffb17ec0a97cd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[92308.503917] RAX: ffffffff82f6ebd1 RBX: ffffb17ec0749068 RCX: 0000000000000000
[92308.505571] RDX: 0000000000000000 RSI: ffffb17ec0749030 RDI: 00000000c000003e
[92308.507677] RBP: ffffb17ec0a97d30 R08: 0000000000000000 R09: 0000000000000000
[92308.509918] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff83a21c80
[92308.511673] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[92308.514356]  ? ___bpf_prog_run+0x851/0x1270
[92308.515945]  ? ___bpf_prog_run+0x851/0x1270
[92308.517570]  ? path_openat+0x725/0x1460
[92308.518877]  ? __bpf_prog_run32+0x33/0x50
[92308.520039]  ? filename_parentat+0x193/0x1b0
[92308.521641]  ? kernel_text_address+0xd4/0xe0
[92308.522801]  ? seccomp_run_filters+0x58/0xb0
[92308.523935]  ? do_filp_open+0xad/0x110
[92308.525740]  ? __seccomp_filter+0x43/0x510
[92308.527803]  ? create_object+0x249/0x300
[92308.529580]  ? __check_object_size+0xb3/0x190
[92308.531520]  ? syscall_trace_enter+0x117/0x2c0
[92308.532975]  ? do_syscall_64+0xf3/0x120
[92308.533612]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[92308.535412] Code: 74 3f 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 41 83 fd 0a 77 32 48 8b 87 98 00 00 00 f6 c4 08 74 06 41 83 fd 08 74 28 <0f> 0b 48 83 c4 08 b8 fe ff ff ff 5b 5d 41 5c 41 5d 41 5e 41 5f 
[92308.542976] ---[ end trace ab4f9128609f842d ]---
  • [ ] 4:
[92308.544840] ------------[ cut here ]------------
[92308.545935] WARNING: CPU: 0 PID: 15856 at /home/user/qtc/release/tempesta/tempesta_fw/http_stream.c:81 tfw_h2_stream_fsm+0x5f/0x250 [tempesta_fw]
[92308.548677] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92308.576894]  [last unloaded: tempesta_lib]
[92308.577910] CPU: 0 PID: 15856 Comm: systemd-journal Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92308.581157] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92308.584468] task: ffff917133d1cb00 task.stack: ffffb17ec0a94000
[92308.586079] RIP: 0010:tfw_h2_stream_fsm+0x5f/0x250 [tempesta_fw]
[92308.588556] RSP: 0018:ffff9171654036f0 EFLAGS: 00010202
[92308.590730] RAX: 0000000000000002 RBX: ffff9170dd4e2680 RCX: 0000000000000001
[92308.593355] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9170dd4e26b0
[92308.594990] RBP: 0000000000000001 R08: ffff917165403744 R09: 0000000000000001
[92308.597271] R10: 00000000fffffffe R11: 000000000000000a R12: ffff9170dd4e26b0
[92308.600091] R13: ffff917165403744 R14: 0000000000000001 R15: 0000000000000001
[92308.602733] FS:  00007f5cd3e7c8c0(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92308.605873] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92308.607523] CR2: 00007f5cd153b000 CR3: 0000000081152001 CR4: 00000000003606f0
[92308.609962] Call Trace:
[92308.610834]  <IRQ>
[92308.611500]  tfw_h2_stream_id_close+0xbd/0x130 [tempesta_fw]
[92308.613525]  tfw_h2_send_resp+0x209/0x560 [tempesta_fw]
[92308.614998]  tfw_http_cli_error_resp_and_log+0x2a6/0x3f0 [tempesta_fw]
[92308.617120]  tfw_http_msg_process_generic+0x24b/0xd10 [tempesta_fw]
[92308.619229]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92308.621348]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92308.623220]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92308.625554]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92308.627792]  ? ttls_recv+0xf2/0x6f0 [tempesta_tls]
[92308.629914]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.631888]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92308.633974]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92308.636239]  ? ip_output+0x71/0xe0
[92308.637665]  ? ip_queue_xmit+0x5c/0x3b0
[92308.638797]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92308.640371]  ? kvm_sched_clock_read+0x1a/0x30
[92308.642042]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92308.643372]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92308.645247]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92308.646535]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92308.648482]  tcp_rcv_established+0x4d2/0x570
[92308.650281]  tcp_v4_do_rcv+0x129/0x1d0
[92308.651848]  tcp_v4_rcv+0x947/0xa50
[92308.653317]  ip_local_deliver_finish+0x9a/0x1c0
[92308.655204]  ip_local_deliver+0x6b/0xe0
[92308.656468]  ? tcp_v4_early_demux+0x112/0x150
[92308.657850]  ? ip_rcv_finish+0x17a/0x400
[92308.659390]  ip_rcv+0x289/0x3c0
[92308.660753]  ? inet_del_offload+0x40/0x40
[92308.662408]  __netif_receive_skb_core+0x84f/0xb30
[92308.664041]  ? tcp_write_xmit+0x2a7/0x11c0
[92308.665785]  ? tfw_wq_pop_ticket+0x7e/0xf0 [tempesta_fw]
[92308.667863]  ? process_backlog+0xa3/0x160
[92308.669028]  process_backlog+0xa3/0x160
[92308.670149]  net_rx_action+0x28e/0x3f0
[92308.671405]  __do_softirq+0x10f/0x2a8
[92308.673343]  irq_exit+0xae/0xb0
[92308.674218]  call_function_single_interrupt+0x7d/0x90
[92308.675138]  </IRQ>
[92308.675643] RIP: 0010:___bpf_prog_run+0x851/0x1270
[92308.676544] RSP: 0018:ffffb17ec0a97cd8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[92308.677923] RAX: ffffffff82f6ebd1 RBX: ffffb17ec0749068 RCX: 0000000000000000
[92308.679128] RDX: 0000000000000000 RSI: ffffb17ec0749030 RDI: 00000000c000003e
[92308.681737] RBP: ffffb17ec0a97d30 R08: 0000000000000000 R09: 0000000000000000
[92308.684932] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff83a21c80
[92308.687591] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[92308.689666]  ? ___bpf_prog_run+0x851/0x1270
[92308.690379]  ? ___bpf_prog_run+0x851/0x1270
[92308.691074]  ? path_openat+0x725/0x1460
[92308.692209]  ? __bpf_prog_run32+0x33/0x50
[92308.692929]  ? filename_parentat+0x193/0x1b0
[92308.693652]  ? kernel_text_address+0xd4/0xe0
[92308.694376]  ? seccomp_run_filters+0x58/0xb0
[92308.695101]  ? do_filp_open+0xad/0x110
[92308.695766]  ? __seccomp_filter+0x43/0x510
[92308.696518]  ? create_object+0x249/0x300
[92308.697268]  ? __check_object_size+0xb3/0x190
[92308.697989]  ? syscall_trace_enter+0x117/0x2c0
[92308.698958]  ? do_syscall_64+0xf3/0x120
[92308.699889]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[92308.700789] Code: c7 c2 45 84 ff 74 46 44 89 f2 83 e2 01 41 f6 c6 01 75 0a 40 80 fd 03 0f 85 f7 01 00 00 8b 43 2c 83 f8 05 7f 42 41 f6 c6 01 74 3c <0f> 0b be 03 00 00 00 4c 89 e7 c6 07 00 0f 1f 40 00 48 83 c4 08 
[92308.705372] ---[ end trace ab4f9128609f842e ]---
  • [ ] 5:
[92315.518389] ------------[ cut here ]------------
[92315.519660] WARNING: CPU: 0 PID: 7 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:678 tfw_huffman_decode+0x400/0x580 [tempesta_fw]
[92315.521791] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[92315.535992]  [last unloaded: tempesta_lib]
[92315.536615] CPU: 0 PID: 7 Comm: ksoftirqd/0 Tainted: G        W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[92315.538757] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[92315.540680] task: ffff917141510f00 task.stack: ffffb17ec0660000
[92315.542231] RIP: 0010:tfw_huffman_decode+0x400/0x580 [tempesta_fw]
[92315.544094] RSP: 0018:ffffb17ec0663660 EFLAGS: 00010286
[92315.545925] RAX: 000000000000034e RBX: ffff9170df80b020 RCX: 0000000000000002
[92315.547085] RDX: 000000000000027f RSI: 000000000000fffa RDI: ffff9170df80b160
[92315.548449] RBP: ffff9170da180788 R08: 0000000000001000 R09: 0000000000000508
[92315.550519] R10: ffffb17ec0663660 R11: 0000000000000001 R12: 00000000fffffffa
[92315.552705] R13: 00000000fffffff9 R14: ffff9170d71cfeb5 R15: 0000000000000000
[92315.554893] FS:  0000000000000000(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[92315.557684] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92315.560548] CR2: 00007f5cd153b000 CR3: 000000000520a005 CR4: 00000000003606f0
[92315.563856] Call Trace:
[92315.564976]  tfw_hpack_decode+0x388/0x1cc0 [tempesta_fw]
[92315.567665]  ? tfw_h2_add_stream+0x5a/0xf0 [tempesta_fw]
[92315.568962]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92315.571487]  tfw_h2_parse_req+0x50/0x180 [tempesta_fw]
[92315.572419]  ? tfw_http_init_parser_req+0x50/0x50 [tempesta_fw]
[92315.573424]  ss_skb_process+0x80/0x140 [tempesta_fw]
[92315.574277]  tfw_http_msg_process_generic+0x1dc/0xd10 [tempesta_fw]
[92315.575998]  ? tfw_h2_conn_terminate_close+0x40/0x40 [tempesta_fw]
[92315.577312]  ? ss_skb_process+0x80/0x140 [tempesta_fw]
[92315.578186]  tfw_h2_frame_process+0x21b/0x490 [tempesta_fw]
[92315.579135]  tfw_http_msg_process+0xb6/0xc0 [tempesta_fw]
[92315.581173]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92315.583108]  tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[92315.585878]  ? lock_timer_base+0x74/0x90
[92315.587503]  tfw_tls_msg_process+0x274/0x440 [tempesta_fw]
[92315.589880]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[92315.591428]  ? kvm_sched_clock_read+0x1a/0x30
[92315.592406]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[92315.593627]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[92315.595792]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[92315.597060]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[92315.598209]  tcp_rcv_established+0x4d2/0x570
[92315.600088]  tcp_v4_do_rcv+0x129/0x1d0
[92315.601105]  tcp_v4_rcv+0x947/0xa50
[92315.601931]  ip_local_deliver_finish+0x9a/0x1c0
[92315.603461]  ip_local_deliver+0x6b/0xe0
[92315.604661]  ? tcp_v4_early_demux+0x112/0x150
[92315.605592]  ? ip_rcv_finish+0x17a/0x400
[92315.607372]  ip_rcv+0x289/0x3c0
[92315.608114]  ? inet_del_offload+0x40/0x40
[92315.609901]  __netif_receive_skb_core+0x84f/0xb30
[92315.611689]  ? process_backlog+0xa3/0x160
[92315.612827]  process_backlog+0xa3/0x160
[92315.613937]  net_rx_action+0x28e/0x3f0
[92315.615406]  __do_softirq+0x10f/0x2a8
[92315.616682]  run_ksoftirqd+0x1c/0x40
[92315.617652]  smpboot_thread_fn+0x10e/0x160
[92315.619315]  kthread+0xff/0x130
[92315.620343]  ? sort_range+0x20/0x20
[92315.621933]  ? kthread_create_on_node+0x70/0x70
[92315.623498]  ret_from_fork+0x35/0x40
[92315.625109] Code: 5c fa ff 48 85 c0 0f 84 45 fe ff ff 48 8b 93 60 01 00 00 48 c7 40 10 01 00 00 00 48 89 10 48 83 83 50 01 00 00 01 e9 e4 fe ff ff <0f> 0b b8 fe ff ff ff e9 1f fe ff ff 48 83 7d 78 00 0f 85 64 fe 

vankoven avatar Mar 04 '20 13:03 vankoven

  • [ ] Also caught when configured Tempesta to stand before Buildbot:
[ 4557.232912] ------------[ cut here ]------------
[ 4557.235488] WARNING: CPU: 0 PID: 0 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:2392 tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[ 4557.239945] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_generic iTCO_wdt iTCO_vendor_support snd_hda_intel ghash_clmulni_intel evdev snd_hda_codec snd_hda_core binfmt_misc pcspkr snd_hwdep serio_raw virtio_gpu snd_pcm ttm snd_timer virtio_balloon sg virtio_console snd drm_kms_helper soundcore lpc_ich mfd_core drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse uhci_hcd ehci_pci ahci ehci_hcd libahci i2c_i801 libata sym53c8xx virtio_pci virtio_ring scsi_transport_spi usbcore usb_common scsi_mod virtio
[ 4557.266558]  [last unloaded: tempesta_lib]
[ 4557.268074] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[ 4557.272649] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[ 4557.276491] task: ffffffff83e12480 task.stack: ffffffff83e00000
[ 4557.278266] RIP: 0010:tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[ 4557.280814] RSP: 0018:ffff917165403748 EFLAGS: 00010293
[ 4557.282119] RAX: 0000000000000003 RBX: 0000000000000000 RCX: 0000000000000032
[ 4557.284592] RDX: ffff917165403890 RSI: 0000000000000032 RDI: 0000000000000035
[ 4557.286068] RBP: 0000000000000000 R08: 0000000000000003 R09: ffff917165403868
[ 4557.287926] R10: 0000000000000003 R11: ffff9171062e803a R12: 0000000000003530
[ 4557.292976] R13: 0000000000007400 R14: ffff91710679d020 R15: ffff917165403828
[ 4557.295560] FS:  0000000000000000(0000) GS:ffff917165400000(0000) knlGS:0000000000000000
[ 4557.297592] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 4557.298691] CR2: 0000557d663629f8 CR3: 000000000520a003 CR4: 00000000003606f0
[ 4557.300134] Call Trace:
[ 4557.300796]  <IRQ>
[ 4557.301283]  ? irq_work_interrupt+0x7d/0x90
[ 4557.302154]  tfw_h2_resp_status_write+0x100/0x250 [tempesta_fw]
[ 4557.303433]  tfw_h2_send_resp+0x150/0x560 [tempesta_fw]
[ 4557.305719]  tfw_http_cli_error_resp_and_log+0x2a6/0x3f0 [tempesta_fw]
[ 4557.308818]  ? kmem_cache_free+0x1a7/0x1d0
[ 4557.310749]  tfw_http_msg_process_generic+0x150/0xd10 [tempesta_fw]
[ 4557.313608]  ? native_safe_halt+0x2/0x10
[ 4557.315383]  ? __kmalloc+0x18c/0x200
[ 4557.317032]  tfw_http_msg_process+0x84/0xc0 [tempesta_fw]
[ 4557.319054]  ? virtqueue_add_outbuf+0x27d/0x30a [virtio_ring]
[ 4557.320701]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[ 4557.326824]  ? minmax_running_min+0x3b/0x90
[ 4557.329635]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[ 4557.331402]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[ 4557.333951]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[ 4557.335301]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[ 4557.336298]  tcp_data_queue+0x4f5/0xc50
[ 4557.337388]  tcp_rcv_established+0x27c/0x570
[ 4557.338845]  ? tcp_v4_inbound_md5_hash+0x62/0x1b0
[ 4557.340617]  tcp_v4_do_rcv+0x129/0x1d0
[ 4557.341928]  tcp_v4_rcv+0x947/0xa50
[ 4557.343604]  ip_local_deliver_finish+0x9a/0x1c0
[ 4557.345648]  ip_local_deliver+0x6b/0xe0
[ 4557.347407]  ? tcp_v4_early_demux+0x112/0x150
[ 4557.349056]  ? ip_rcv_finish+0x17a/0x400
[ 4557.350862]  ip_rcv+0x289/0x3c0
[ 4557.351874]  ? inet_del_offload+0x40/0x40
[ 4557.352605]  __netif_receive_skb_core+0x84f/0xb30
[ 4557.354041]  ? process_backlog+0xa3/0x160
[ 4557.358165]  process_backlog+0xa3/0x160
[ 4557.360881]  net_rx_action+0x28e/0x3f0
[ 4557.361923]  __do_softirq+0x10f/0x2a8
[ 4557.362642]  irq_exit+0xae/0xb0
[ 4557.363260]  call_function_single_interrupt+0x7d/0x90
[ 4557.365250]  </IRQ>
[ 4557.365937] RIP: 0010:native_safe_halt+0x2/0x10
[ 4557.366983] RSP: 0018:ffffffff83e03e98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[ 4557.369610] RAX: ffffffff8349a7a0 RBX: ffffffff83e12480 RCX: 0000000000000000
[ 4557.372141] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 4557.375379] RBP: 0000000000000000 R08: 0000048deed6578e R09: ffff91713e9c4300
[ 4557.378307] R10: 0000000000000000 R11: 000004f26258ebfb R12: ffffffff83e12480
[ 4557.381539] R13: ffffffff83e12480 R14: 0000000000000000 R15: 0000000000000000
[ 4557.384942]  ? __sched_text_end+0x3/0x3
[ 4557.385566]  default_idle+0x1a/0xf0
[ 4557.386384]  do_idle+0x16e/0x1f0
[ 4557.387427]  cpu_startup_entry+0x6f/0x80
[ 4557.389219]  start_kernel+0x467/0x487
[ 4557.392020]  secondary_startup_64+0xa5/0xb0
[ 4557.393435] Code: fb 7f 48 8d 42 05 0f 86 7b ff ff ff 89 de 48 c1 eb 07 48 83 c0 01 83 ce 80 83 c1 01 48 83 fb 7f 40 88 70 ff 77 e6 e9 5c ff ff ff <0f> 0b 48 8b 44 24 58 e9 08 f7 ff ff 41 89 dc e9 9a f0 ff ff 0f 

vankoven avatar Mar 04 '20 14:03 vankoven

  • [ ] One more warning:
[78406.287899] ------------[ cut here ]------------
[78406.289256] WARNING: CPU: 1 PID: 584 at /home/user/qtc/release/tempesta/tempesta_fw/hpack.c:2391 tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[78406.291874] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) sha256_ssse3 sha512_ssse3 sha512_generic ccm iTCO_wdt iTCO_vendor_support kvm_intel snd_hda_codec_generic binfmt_misc kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_intel snd_hda_codec snd_hda_core ghash_clmulni_intel snd_hwdep snd_pcm snd_timer evdev serio_raw pcspkr snd soundcore virtio_balloon sg virtio_gpu virtio_console ttm lpc_ich mfd_core drm_kms_helper drm shpchp button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto ecb aesni_intel crypto_simd cryptd glue_helper aes_x86_64 crc32c_generic sr_mod cdrom virtio_blk virtio_net crc32c_intel psmouse ahci libahci sym53c8xx scsi_transport_spi uhci_hcd ehci_pci ehci_hcd libata i2c_i801 scsi_mod usbcore virtio_pci usb_common virtio_ring virtio
[78406.317135]  [last unloaded: tempesta_lib]
[78406.319725] CPU: 1 PID: 584 Comm: nginx Tainted: G    B   W  O    4.14.0-tempesta-kmemleak-amd64 #1 Debian 4.14.32-tfw7-1
[78406.327777] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS ?-20191223_100556-anatol 04/01/2014
[78406.330875] task: ffff9c40fcd9e900 task.stack: ffffbab680db8000
[78406.332466] RIP: 0010:tfw_hpack_encode+0x127f/0x12a0 [tempesta_fw]
[78406.336057] RSP: 0018:ffff9c41655037b0 EFLAGS: 00010293
[78406.337360] RAX: 000000000000001f RBX: 0000000000000000 RCX: 0000000000000029
[78406.339593] RDX: ffff9c41655240bf RSI: 0000000000000029 RDI: 000000000000002e
[78406.341660] RBP: 0000000000000000 R08: 0000000000000003 R09: ffff9c41655038f0
[78406.346238] R10: 000000000000001f R11: ffff9c413e1f61ee R12: 0000000000002e30
[78406.348124] R13: 0000000000002e00 R14: 0000000000000002 R15: ffff9c4165503890
[78406.351018] FS:  00007fde01a1c740(0000) GS:ffff9c4165500000(0000) knlGS:0000000000000000
[78406.356923] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[78406.363536] CR2: 000055aa84341ec8 CR3: 000000007cee8002 CR4: 00000000003606e0
[78406.369647] Call Trace:
[78406.371385]  <IRQ>
[78406.372800]  tfw_h2_add_hdr_via+0xef/0x140 [tempesta_fw]
[78406.374991]  tfw_http_resp_cache_cb+0x4ac/0x5f0 [tempesta_fw]
[78406.376532]  ? __tfw_http_msg_add_str_data+0x5e/0x100 [tempesta_fw]
[78406.378624]  ? tfw_http_conn_drop+0x280/0x280 [tempesta_fw]
[78406.380504]  tfw_cache_process+0x72/0x2a0 [tempesta_fw]
[78406.385721]  ? tfw_gfsm_move+0x14b/0x180 [tempesta_fw]
[78406.393488]  ? kvm_clock_get_cycles+0x1a/0x20
[78406.397727]  tfw_http_resp_cache+0xe1/0x150 [tempesta_fw]
[78406.401197]  tfw_http_msg_process_generic+0x893/0xd10 [tempesta_fw]
[78406.406241]  ? ip_finish_output2+0x2be/0x3b0
[78406.408679]  ? pg_skb_alloc+0x253/0x470
[78406.410788]  tfw_http_msg_process+0x84/0xc0 [tempesta_fw]
[78406.413430]  __gfsm_fsm_exec+0x56/0x90 [tempesta_fw]
[78406.415835]  ? consume_skb+0x27/0x80
[78406.418778]  tfw_connection_recv+0x41/0x60 [tempesta_fw]
[78406.424415]  ? tfw_connection_send+0x30/0x30 [tempesta_fw]
[78406.428091]  ss_tcp_process_data+0x1ea/0x480 [tempesta_fw]
[78406.440355]  ? mod_timer+0xb0/0x3b0
[78406.441168]  ss_tcp_data_ready+0x43/0x90 [tempesta_fw]
[78406.442478]  tcp_rcv_established+0x4d2/0x570
[78406.443676]  tcp_v4_do_rcv+0x129/0x1d0
[78406.444619]  tcp_v4_rcv+0x947/0xa50
[78406.446035]  ip_local_deliver_finish+0x9a/0x1c0
[78406.447149]  ip_local_deliver+0x6b/0xe0
[78406.448230]  ? nf_hook_slow+0x43/0xc0
[78406.449447]  ip_rcv+0x289/0x3c0
[78406.450441]  ? inet_del_offload+0x40/0x40
[78406.451605]  __netif_receive_skb_core+0x84f/0xb30
[78406.453209]  ? tfw_wq_pop_ticket+0x7e/0xf0 [tempesta_fw]
[78406.456620]  ? process_backlog+0xa3/0x160
[78406.464121]  process_backlog+0xa3/0x160
[78406.467997]  net_rx_action+0x28e/0x3f0
[78406.471433]  __do_softirq+0x10f/0x2a8
[78406.473576]  do_softirq_own_stack+0x2a/0x40
[78406.476495]  </IRQ>
[78406.477463]  do_softirq.part.17+0x3d/0x50
[78406.479665]  __local_bh_enable_ip+0x55/0x60
[78406.480820]  ip_finish_output2+0x1b3/0x3b0
[78406.483415]  ? ip_output+0x71/0xe0
[78406.484672]  ip_output+0x71/0xe0
[78406.486736]  ? ip_queue_xmit+0x5c/0x3b0
[78406.489085]  tcp_transmit_skb+0x535/0x9c0
[78406.490854]  tcp_write_xmit+0x223/0x11c0
[78406.492331]  ? _copy_from_iter_full+0x93/0x230
[78406.493030]  __tcp_push_pending_frames+0x2d/0xd0
[78406.494068]  tcp_sendmsg_locked+0x68e/0xe60
[78406.495932]  tcp_sendmsg+0x27/0x40
[78406.497243]  sock_sendmsg+0x36/0x40
[78406.498259]  sock_write_iter+0x87/0x100
[78406.499226]  do_iter_readv_writev+0x11b/0x190
[78406.501233]  do_iter_write+0x80/0x190
[78406.502836]  vfs_writev+0x98/0x110
[78406.504009]  ? do_writev+0x5e/0xf0
[78406.505266]  do_writev+0x5e/0xf0
[78406.507133]  ? SyS_pread64+0x90/0xb0
[78406.509205]  do_syscall_64+0x8d/0x120
[78406.510761]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[78406.512833] RIP: 0033:0x7fde00354fe4
[78406.513707] RSP: 002b:00007ffcac2496f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000014
[78406.517015] RAX: ffffffffffffffda RBX: 00007fde019f02c8 RCX: 00007fde00354fe4
[78406.520071] RDX: 0000000000000001 RSI: 00007ffcac2497b0 RDI: 000000000000000b
[78406.526661] RBP: 000055bf73ed0cb8 R08: 000055bf73f4e460 R09: 00007ffcac2497b0
[78406.530225] R10: 000055bf73eb2d50 R11: 0000000000000246 R12: 00007ffcac249790
[78406.536453] R13: 000055bf7221548c R14: 000000007fffefff R15: 0000000000005db7
[78406.538593] Code: fb 7f 48 8d 42 05 0f 86 7b ff ff ff 89 de 48 c1 eb 07 48 83 c0 01 83 ce 80 83 c1 01 48 83 fb 7f 40 88 70 ff 77 e6 e9 5c ff ff ff <0f> 0b 48 8b 44 24 58 e9 08 f7 ff ff 41 89 dc e9 9a f0 ff ff 0f 
[78406.546574] ---[ end trace d2f49a8c8d368827 ]---

vankoven avatar Apr 10 '20 09:04 vankoven

  • [x] Just tried to load our web site trough Tempesta FW current master as of date of the message using Firefox browser and got this warning:
[129225.889634] ------------[ cut here ]------------
[129225.892203] WARNING: CPU: 0 PID: 3257 at /root/tempesta/tempesta_fw/hpack.c:2401 tfw_hpack_node_compare+0xc08/0xc30 [tempesta_fw]
[129225.896297] Modules linked in: tempesta_fw(O) tempesta_db(O) tempesta_tls(O) tempesta_lib(O) tcp_diag inet_diag bochs_drm ttm crct10dif_pclmul drm_kms_helper ppdev drm sg button parport_pc parport fb_sys_fops syscopyarea sysfillrect sysimgblt serio_raw binfmt_misc ip_tables x_tables ext4 crc16 mbcache jbd2 fscrypto sr_mod sd_mod cdrom ata_generic virtio_net ata_piix virtio_pci virtio_ring virtio psmouse libata i2c_piix4 scsi_mod [last unloaded: tempesta_lib]
[129225.899082] CPU: 0 PID: 3257 Comm: nginx Tainted: G           O    4.14.32-kdump+ #158
[129225.899082] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-2.fc32 04/01/2014
[129225.899082] task: ffff9a3cdc268140 task.stack: ffff9a3cd6290000
[129225.899082] RIP: 0010:tfw_hpack_node_compare+0xc08/0xc30 [tempesta_fw]
[129225.899082] RSP: 0018:ffff9a3cffc03718 EFLAGS: 00010293
[129225.899082] RAX: ffff9a3cb377816e RBX: 000000000000001f RCX: 0000000000000029
[129225.899082] RDX: ffff9a3cffc227ff RSI: 0000000000000029 RDI: ffff9a3cffc03848
[129225.899082] RBP: 000000000000001f R08: 0000000000000029 R09: 0000000000002e30
[129225.899082] R10: 0000000000000003 R11: ffff9a3cffc038a8 R12: 0000000000000000
[129225.899082] R13: 0000000000002e30 R14: 0000000000002030 R15: 0000000000000000
[129225.899082] FS:  00007efd12e5ab80(0000) GS:ffff9a3cffc00000(0000) knlGS:0000000000000000
[129225.899082] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[129225.899082] CR2: 000056294c2cb1c8 CR3: 000000005c32e001 CR4: 00000000003606f0
[129225.899082] Call Trace:
[129225.899082]  <IRQ>
[129225.899082]  tfw_hpack_rbtree_find.isra.1+0x4a/0x100 [tempesta_fw]
[129225.899082]  tfw_hpack_encode+0x13d/0x570 [tempesta_fw]
[129225.899082]  tfw_h2_add_hdr_via+0xea/0x130 [tempesta_fw]
[129225.899082]  tfw_http_resp_cache_cb+0x4a9/0x5f0 [tempesta_fw]
[129225.899082]  ? tfw_http_conn_drop+0x250/0x250 [tempesta_fw]
[129225.899082]  tfw_cache_process+0x6d/0x290 [tempesta_fw]
[129225.899082]  ? tfw_gfsm_move+0x146/0x180 [tempesta_fw]
[129225.899082]  ? efi_native_runtime_setup+0xa0/0xa0
[129225.899082]  tfw_http_resp_cache+0x141/0x180 [tempesta_fw]
[129225.899082]  tfw_http_msg_process_generic+0x839/0xce0 [tempesta_fw]
[129225.899082]  ? dev_hard_start_xmit+0x9e/0x1e0
[129225.899082]  ? pg_skb_alloc+0x223/0x420
[129225.899082]  tfw_http_msg_process+0x7f/0xc0 [tempesta_fw]
[129225.899082]  __gfsm_fsm_exec+0x51/0x90 [tempesta_fw]
[129225.899082]  ? consume_skb+0x22/0x80
[129225.899082]  tfw_connection_recv+0x3c/0x60 [tempesta_fw]
[129225.899082]  ? tfw_connection_send+0x20/0x20 [tempesta_fw]
[129225.899082]  ss_tcp_process_data+0x1e5/0x480 [tempesta_fw]
[129225.899082]  ss_tcp_data_ready+0x3e/0x90 [tempesta_fw]
[129225.899082]  tcp_data_queue+0x4f0/0xc50
[129225.899082]  tcp_rcv_established+0x277/0x570
[129225.899082]  tcp_v4_do_rcv+0x124/0x1c0
[129225.899082]  tcp_v4_rcv+0x93f/0xa40
[129225.899082]  ip_local_deliver_finish+0x95/0x1c0

Firefox didn't show the header image Screen Shot 2020-11-17 at 23 11 15-fullpage

Tempesta config:

listen 192.168.100.4:443 proto=h2;
listen 192.168.100.4:80;

frang_limits {
	client_header_timeout 20;
	client_body_timeout 10;
	http_header_chunk_cnt 10;
	http_body_chunk_cnt 0;
	http_methods GET;
	http_uri_len 512;
	http_resp_code_block 400 403 404 3 10;
}

# Allow only following characters in URI (no '%'): /a-zA-Z0-9&?:-._=
http_uri_brange 0x2f 0x41-0x5a 0x61-0x7a 0x30-0x39 0x26 0x3f 0x3a 0x2d 0x2e 0x5f 0x3d;

block_action attack reply;
block_action error reply;

srv_group default {
	server 127.0.0.1:8080 conns_n=4;
}

vhost debian {
	tls_certificate /root/tempesta/etc/tfw-root.crt;
	tls_certificate_key /root/tempesta/etc/tfw-root.key;

	resp_hdr_set Strict-Transport-Security "max-age=31536000; includeSubDomains";

	proxy_pass default;
}

cache 0;
#cache_fulfill * *;

http_chain {
	-> debian;
}

krizhanovsky avatar Nov 17 '20 20:11 krizhanovsky

Warnings on tfw_http_msg_process_generic() and tfw_http_msg_hdr_close() warnings raised in https://github.com/tempesta-tech/tempesta-test/pull/254

krizhanovsky avatar Jul 26 '22 13:07 krizhanovsky

Warnings for tfw_http_msg_process_generic are covered by the disabled test https://github.com/tempesta-tech/tempesta-test/pull/296/files

krizhanovsky avatar Aug 25 '22 18:08 krizhanovsky

Found some more:

  • [x] ./h2spec http2/8.1.2.1/3 -h f35tfw.local -p 443 -t -k
[tempesta fw] Warning: failed to parse request: 192.168.122.1
------------[ cut here ]------------
WARNING: CPU: 2 PID: 0 at /root/host/tempesta/fw/http.c:5053 tfw_http_cli_error_resp_and_log+0x285/0xb80 [tempesta_fw]
Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJEC
T nf_reject_ipv4 ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_f
ilter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support input_leds led_class irqbypass i2c_i801 i2c_smbus
pcspkr virtiofs lpc_ich virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_scsi serio_raw virtio_net virt
io_blk net_failover virtio_console failover qemu_fw_cfg ipmi_devintf ipmi_msghandler [last unloaded: tempesta_lib]
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G    B   W  OE     5.10.35+ #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:tfw_http_cli_error_resp_and_log+0x285/0xb80 [tempesta_fw]
Code: 00 48 8d 54 24 40 31 f6 48 89 54 24 10 48 83 c7 14 e8 bf 7d fb ff 48 8b 54 24 10 4c 89 ee 48 c7 c7 00 e3 d9 c0 e8 f4 57 8e c2 <0f> 0b e9 3e fe ff ff 80 3
c 02 00 0f 85 d8 06 00 00 49 8b 87 a0 00
RSP: 0018:ffffc900001f7e08 EFLAGS: 00010286
RAX: 000000000000003d RBX: 1ffff9200003efc5 RCX: 0000000000000000
RDX: 1ffffffff08507a0 RSI: 00000000000000f6 RDI: fffff5200003efb8
RBP: 0000000000000000 R08: 000000000000003d R09: ffffffff84cdcf87
R10: fffffbfff099b9f0 R11: 0000000000000001 R12: 0000000000000001
R13: ffffffffc0da07e0 R14: 0000000000000001 R15: ffff88801dd2a020
FS:  0000000000000000(0000) GS:ffff888149700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc68afc9018 CR3: 000000012c71e003 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <IRQ>
 ? tfw_h1_send_err_resp+0xe0/0xe0 [tempesta_fw]
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_http_req_process+0x938/0x15a0 [tempesta_fw]
 ? tfw_http_req_cache_cb+0x17e0/0x17e0 [tempesta_fw]
 ? dynamic_emit_prefix+0x350/0x350
 ? __dynamic_pr_debug+0xeb/0x130
 ? dynamic_emit_prefix+0x2c0/0x350
 ? vprintk_emit+0x1c6/0x310
 tfw_http_msg_process_generic+0x883/0x1380 [tempesta_fw]
 ? ss_skb_chop_head_tail+0x1f9/0x790 [tempesta_fw]
 ? tfw_h2_frame_local_resp+0x90/0x90 [tempesta_fw]
 ? ss_skb_expand_head_tail+0x2e0/0x2e0 [tempesta_fw]
 ? ttls_send_alert+0x1d0/0x1d0 [tempesta_tls]
 ? rcu_nmi_exit+0x50/0xc0
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_h2_frame_process+0x3c1/0x1280 [tempesta_fw]
 ? tfw_h2_stream_id_close+0x380/0x380 [tempesta_fw]
 tfw_connection_recv+0x102/0x1e0 [tempesta_fw]
 tfw_tls_connection_recv+0x52b/0xaa0 [tempesta_fw]
 ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]
 ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]
 ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]
 ss_tcp_data_ready+0x141/0x280 [tempesta_fw]
 tcp_rcv_established+0x1155/0x1d10
 ? tcp_data_queue+0x4cb0/0x4cb0
 tcp_v4_do_rcv+0x4fa/0x750
 tcp_v4_rcv+0x2588/0x3530
 ? tcp_v4_early_demux+0x790/0x790
 ip_protocol_deliver_rcu+0x6a/0x550
 ? nf_confirm+0x340/0x340 [nf_conntrack]
 ip_local_deliver_finish+0x1a4/0x250
 ip_local_deliver+0x246/0x2a0
 ? ip_local_deliver_finish+0x250/0x250
 ? ip_route_input_slow+0x32a0/0x32a0
 ? ip_protocol_deliver_rcu+0x550/0x550
 ip_sublist_rcv_finish+0x11f/0x1b0
 ip_sublist_rcv+0x3e5/0x630
 ? ip_rcv_core+0xa70/0xa70
 ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0
 ? get_page_from_freelist+0x1b39/0x4da0
 ip_list_rcv+0x293/0x3b0
 ? ip_rcv+0x180/0x180
 __netif_receive_skb_list_core+0x61d/0x8b0
 ? page_to_skb+0x69b/0x9f0 [virtio_net]
 ? __netif_receive_skb_core.constprop.0+0x3050/0x3050
 ? __zone_watermark_ok+0x420/0x420
 ? virtqueue_add_split+0x4f0/0x1120
 netif_receive_skb_list_internal+0x5d7/0xc10
 ? __netif_receive_skb_list_core+0x8b0/0x8b0
 ? virtqueue_add_inbuf_ctx+0xc2/0xf0
 ? virtqueue_add_sgs+0x120/0x120
 ? try_fill_recv+0xaac/0x1c30 [virtio_net]
 napi_complete_done+0x188/0x5d0
 virtqueue_napi_complete+0x2b/0x80 [virtio_net]
 virtnet_poll+0x90c/0xc81 [virtio_net]
 ? receive_buf+0x57f0/0x57f0 [virtio_net]
 ? kmem_cache_free_bulk+0x39d/0x7f0
 net_rx_action+0x37c/0xbc0
 ? napi_complete_done+0x5d0/0x5d0
 ? sched_clock_cpu+0x18/0x160
 __do_softirq+0x1a6/0x5ab
 ? handle_edge_irq+0x35e/0xb60
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 do_softirq_own_stack+0x37/0x40
 irq_exit_rcu+0x1ad/0x260
 common_interrupt+0x71/0x110
 asm_common_interrupt+0x1e/0x40
RIP: 0010:native_safe_halt+0xe/0x10
Code: ff ff ff 4c 89 e7 e8 b1 29 2b fe e9 ed fe ff ff 48 89 ef e8 a4 29 2b fe eb a3 cc cc e9 07 00 00 00 0f 00 2d c4 f7 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 0
0 2d b4 f7 4d 00 f4 c3 cc cc 0f 1f 44 00
RSP: 0018:ffffc90000127e28 EFLAGS: 00000206
RAX: ffffffff835702f0 RBX: ffff888120ab5080 RCX: ffffffff8354ee5d
RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff888149732f80
RBP: ffffed1024156a10 R08: 0000000000000001 R09: ffff888149732f83
R10: ffffed10292e65f0 R11: 0000000000000001 R12: 0000000000000002
R13: ffffffff84ce0e20 R14: 0000000000000000 R15: dffffc0000000000
 ? __sched_text_end+0x7/0x7
 ? rcu_eqs_enter.constprop.0+0x7d/0xa0
 default_idle+0xa/0x10
 default_idle_call+0x74/0x1f0
 do_idle+0x3bd/0x550
 ? arch_cpu_idle_exit+0x40/0x40
 cpu_startup_entry+0x19/0x20
 start_secondary+0x22b/0x2d0
 ? set_cpu_sibling_map+0x21e0/0x21e0
 ? start_cpu0+0xc/0xc
 secondary_startup_64_no_verify+0xc2/0xcb
---[ end trace 321d13b12076d19e ]---
  • [x] ./h2spec http2/8.1.2.1/3 -h f35tfw.local -p 443 -t -k
[tempesta fw]       enter tfw_h2_stream_fsm: stream->state=2, stream->id=1, type=1, flags=0x1
------------[ cut here ]------------
WARNING: CPU: 2 PID: 0 at /root/host/tempesta/fw/http_stream.c:80 tfw_h2_stream_fsm+0x4b7/0x770 [tempesta_fw]
Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJEC
T nf_reject_ipv4 ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_f
ilter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support input_leds led_class irqbypass i2c_i801 i2c_smbus
pcspkr virtiofs lpc_ich virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_scsi serio_raw virtio_net virt
io_blk net_failover virtio_console failover qemu_fw_cfg ipmi_devintf ipmi_msghandler [last unloaded: tempesta_lib]
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G    B   W  OE     5.10.35+ #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
RIP: 0010:tfw_h2_stream_fsm+0x4b7/0x770 [tempesta_fw]
Code: ea 03 0f b6 14 02 4c 89 d0 83 e0 07 83 c0 03 38 d0 7c 08 84 d2 0f 85 65 02 00 00 c7 43 2c 02 00 00 00 45 31 ed e9 ac fd ff ff <0f> 0b e9 9f fd ff ff 83 e
5 05 45 31 ed 40 80 fd 04 0f 84 95 fd ff
RSP: 0018:ffffc900001f7928 EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffff88807228c9c0 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 00000000000000f6 RDI: fffff5200003eef2
RBP: 0000000000000001 R08: 000000000000005d R09: ffffffff84cdcf87
R10: ffff88807228c9ec R11: 0000000000000001 R12: 0000000000000001
R13: 0000000000000001 R14: ffff88807228c9f0 R15: ffffc900001f79b0
FS:  0000000000000000(0000) GS:ffff888149700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc68afc9018 CR3: 000000012c71e003 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <IRQ>
 ? _raw_write_lock_irqsave+0xc0/0xc0
 tfw_h2_stream_id_close+0x161/0x380 [tempesta_fw]
 ? tfw_h2_stream_id+0xc0/0xc0 [tempesta_fw]
 ? __zone_watermark_ok+0x420/0x420
 tfw_h2_prep_resp+0xfd/0x7a0 [tempesta_fw]
 ? __alloc_pages_nodemask+0x2bd/0x5e0
 ? tfw_h2_resp_status_write+0x2c0/0x2c0 [tempesta_fw]
 ? alloc_pages_current+0x132/0x210
 ? memset+0x20/0x40
 ? __tfw_http_msg_alloc+0x362/0x6e0 [tempesta_fw]
 tfw_h2_send_resp+0x7b/0x130 [tempesta_fw]
 tfw_h2_send_err_resp+0xcf/0xf0 [tempesta_fw]
 ? tfw_h2_send_resp+0x130/0x130 [tempesta_fw]
 ? tfw_http_cli_error_resp_and_log+0x287/0xb80 [tempesta_fw]
 tfw_http_cli_error_resp_and_log+0x87f/0xb80 [tempesta_fw]
 ? tfw_h1_send_err_resp+0xe0/0xe0 [tempesta_fw]
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_http_req_process+0x938/0x15a0 [tempesta_fw]
 ? tfw_http_req_cache_cb+0x17e0/0x17e0 [tempesta_fw]
 ? dynamic_emit_prefix+0x350/0x350
 ? __dynamic_pr_debug+0xeb/0x130
 ? dynamic_emit_prefix+0x2c0/0x350
 ? vprintk_emit+0x1c6/0x310
 tfw_http_msg_process_generic+0x883/0x1380 [tempesta_fw]
 ? ss_skb_chop_head_tail+0x1f9/0x790 [tempesta_fw]
 ? tfw_h2_frame_local_resp+0x90/0x90 [tempesta_fw]
 ? ss_skb_expand_head_tail+0x2e0/0x2e0 [tempesta_fw]
 ? ttls_send_alert+0x1d0/0x1d0 [tempesta_tls]
 ? rcu_nmi_exit+0x50/0xc0
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
 tfw_h2_frame_process+0x3c1/0x1280 [tempesta_fw]
 ? tfw_h2_stream_id_close+0x380/0x380 [tempesta_fw]
 tfw_connection_recv+0x102/0x1e0 [tempesta_fw]
 tfw_tls_connection_recv+0x52b/0xaa0 [tempesta_fw]
 ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]
 ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]
 ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]
 ss_tcp_data_ready+0x141/0x280 [tempesta_fw]
 tcp_rcv_established+0x1155/0x1d10
 ? tcp_data_queue+0x4cb0/0x4cb0
 tcp_v4_do_rcv+0x4fa/0x750
 tcp_v4_rcv+0x2588/0x3530
 ? tcp_v4_early_demux+0x790/0x790
 ip_protocol_deliver_rcu+0x6a/0x550
 ? nf_confirm+0x340/0x340 [nf_conntrack]
 ip_local_deliver_finish+0x1a4/0x250
 ip_local_deliver+0x246/0x2a0
 ? ip_local_deliver_finish+0x250/0x250
 ? ip_route_input_slow+0x32a0/0x32a0
 ? ip_protocol_deliver_rcu+0x550/0x550
 ip_sublist_rcv_finish+0x11f/0x1b0
 ip_sublist_rcv+0x3e5/0x630
 ? ip_rcv_core+0xa70/0xa70
 ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0
 ? get_page_from_freelist+0x1b39/0x4da0
 ip_list_rcv+0x293/0x3b0
 ? ip_rcv+0x180/0x180
 __netif_receive_skb_list_core+0x61d/0x8b0
 ? page_to_skb+0x69b/0x9f0 [virtio_net]
 ? __netif_receive_skb_core.constprop.0+0x3050/0x3050
 ? __zone_watermark_ok+0x420/0x420
 ? virtqueue_add_split+0x4f0/0x1120
 netif_receive_skb_list_internal+0x5d7/0xc10
 ? __netif_receive_skb_list_core+0x8b0/0x8b0
 ? virtqueue_add_inbuf_ctx+0xc2/0xf0
 ? virtqueue_add_sgs+0x120/0x120
 ? try_fill_recv+0xaac/0x1c30 [virtio_net]
 napi_complete_done+0x188/0x5d0
 virtqueue_napi_complete+0x2b/0x80 [virtio_net]
 virtnet_poll+0x90c/0xc81 [virtio_net]
 ? receive_buf+0x57f0/0x57f0 [virtio_net]
 ? kmem_cache_free_bulk+0x39d/0x7f0
 net_rx_action+0x37c/0xbc0
 ? napi_complete_done+0x5d0/0x5d0
 ? sched_clock_cpu+0x18/0x160
 __do_softirq+0x1a6/0x5ab
 ? handle_edge_irq+0x35e/0xb60
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 do_softirq_own_stack+0x37/0x40
 irq_exit_rcu+0x1ad/0x260
 common_interrupt+0x71/0x110
 asm_common_interrupt+0x1e/0x40
RIP: 0010:native_safe_halt+0xe/0x10
Code: ff ff ff 4c 89 e7 e8 b1 29 2b fe e9 ed fe ff ff 48 89 ef e8 a4 29 2b fe eb a3 cc cc e9 07 00 00 00 0f 00 2d c4 f7 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 0
0 2d b4 f7 4d 00 f4 c3 cc cc 0f 1f 44 00
RSP: 0018:ffffc90000127e28 EFLAGS: 00000206
RAX: ffffffff835702f0 RBX: ffff888120ab5080 RCX: ffffffff8354ee5d
RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff888149732f80
RBP: ffffed1024156a10 R08: 0000000000000001 R09: ffff888149732f83
R10: ffffed10292e65f0 R11: 0000000000000001 R12: 0000000000000002
R13: ffffffff84ce0e20 R14: 0000000000000000 R15: dffffc0000000000
 ? __sched_text_end+0x7/0x7
 ? rcu_eqs_enter.constprop.0+0x7d/0xa0
 default_idle+0xa/0x10
 default_idle_call+0x74/0x1f0
 do_idle+0x3bd/0x550
 ? arch_cpu_idle_exit+0x40/0x40
 cpu_startup_entry+0x19/0x20
 start_secondary+0x22b/0x2d0
 ? set_cpu_sibling_map+0x21e0/0x21e0
 ? start_cpu0+0xc/0xc
 secondary_startup_64_no_verify+0xc2/0xcb
---[ end trace 321d13b12076d19f ]---
[tempesta fw]       exit tfw_h2_stream_fsm: stream->state=2, res=3
[tempesta fw]   tfw_h2_send_resp: HTTP/2 response message transformation error: conn=[0000000067ae9791]

s0nx avatar Sep 28 '22 17:09 s0nx

I can't reproduce this one, but with the same config i've caught another warning.

[tempesta tls] Warning: [::ffff:192.168.122.1] ClientHello: bad extension size 281 (remaining extensions size 285)                                             
------------[ cut here ]------------                                                                                                                                                                                                                                                                                           
WARNING: CPU: 2 PID: 0 at /root/host/tempesta/tls/tls_srv.c:916 ttls_parse_client_hello.cold+0xeec/0x1012 [tempesta_tls]                                                                                                                                                                                                       
Modules linked in: tempesta_fw(OE) tempesta_db(OE) tempesta_tls(OE) tempesta_lib(OE) sha256_ssse3 sha512_ssse3 xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support input_leds led_class irqbypass i2c_i801 i2c_smbus pcspkr virtiofs lpc_ich virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_scsi serio_raw virtio_net virtio_
blk net_failover virtio_console failover qemu_fw_cfg ipmi_devintf ipmi_msghandler [last unloaded: tempesta_lib]                                                
CPU: 2 PID: 0 Comm: swapper/2 Tainted: G    B   W  OE     5.10.35+ #1                                                                                                                                                                                                                                                          
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015                                                                                      
RIP: 0010:ttls_parse_client_hello.cold+0xeec/0x1012 [tempesta_tls]                                                                                                                                                                                                                                                             
Code: 0f 0b e9 b7 f9 ff ff 89 14 24 e8 6a ea e8 c0 8b 14 24 e9 70 f8 ff ff 48 8d 75 38 44 89 e1 48 c7 c7 80 07 9a c0 e8 68 1a b0 c2 <0f> 0b e9 77 f1 ff ff 49 8d 74 24 38 48 c7 c7 20 26 9a c0 e8 50 1a
RSP: 0018:ffffc900001f7c40 EFLAGS: 00010286                                                                                                                    
RAX: 0000000000000072 RBX: 0000000000000119 RCX: 0000000000000000                                                                                              
RDX: 0000000000000027 RSI: 0000000000000004 RDI: fffff5200003ef7f                                                                                              
RBP: ffff88817eb60a00 R08: 0000000000000072 R09: ffff888149720a8b                                                                                              
R10: ffffed10292e4151 R11: 0000000000000001 R12: 000000000000011d                                                                                              
R13: ffff888037ad5bbf R14: 000000000000014c R15: ffff888128b014f0                                                                                              
FS:  0000000000000000(0000) GS:ffff888149700000(0000) knlGS:0000000000000000                                                                                   
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                                                                                              
CR2: 00007fc687e84660 CR3: 000000012c71e006 CR4: 0000000000770ee0                                                                                                                                                                                                                                                              
PKRU: 55555554                                                                                                                                                                                                                                                                                                                 
Call Trace:                                                                                                                                                                                                                                                                                                                    
 <IRQ>                                                                                                                                                                                                                                                                                                                         
 ? rgb_background+0xe2/0x140                                                                                                                                                                                                                                                                                                   
 ? desc_read_finalized_seq+0x100/0x100                                                                                                                         
 ? atomic_notifier_call_chain+0xa8/0xe0                                                                                                                        
 ttls_handshake_server_step+0xc19/0x41d0 [tempesta_tls]                                                                                                        
 ? prb_read_valid+0x56/0x70                                                                                                                                                                                                                                                                                                    
 ? ttls_write_server_key_exchange+0xf70/0xf70 [tempesta_tls]                                                                                                   
 ? up+0x17/0xc0                                                                                                                                                
 ? vprintk_store+0x1d3/0x420                                                                                                                                                                                                                                                                                                   
 ? _raw_spin_lock+0x6a/0xb0                                                                                                                                    
 ? irq_work_claim+0x16/0x60                                                                                                                                    
 ? irq_work_queue+0xb/0x30                                                                                                                                     
 ? vprintk_emit+0x1c6/0x310                                                                                                                                    
 ? printk+0x96/0xb2                                                                                                                                            
 ? record_print_text.cold+0x11/0x11                                                                                                                            
 ? hex_dump_to_buffer+0xa90/0xa90                                                                                                                                                                                                                                                                                              
 ? __dynamic_pr_debug+0xeb/0x130                                                                                                                               
 ? __dynamic_pr_debug+0xeb/0x130                                                                                                                               
 ? dynamic_emit_prefix+0x350/0x350                                                                                                                             
 ? nowork+0xc/0xc [sha512_ssse3]                                                                                                                               
 ? memcpy+0x39/0x60                                                                                                                                            
 ? sha512_update+0xcc/0x1c0 [sha512_ssse3]                                                                                                                                                                                                                                                                                     
 ? ttls_update_checksum+0x1b6/0x430 [tempesta_tls]                                                                                                             
 ttls_recv+0xb08/0x1e40 [tempesta_tls]                                                                                                                         
 ss_skb_process+0x393/0x570 [tempesta_fw]                                                                                                                      
 ? ttls_handle_alert+0x1b0/0x1b0 [tempesta_tls]                                                                                                                                                                                                                                                                                
 ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]                                                                                                                                                                                                                                                                                   
 ? _raw_spin_lock+0x6a/0xb0                                                                                                                                    
 ? _raw_write_lock_irqsave+0xc0/0xc0                                                                                                                           
 ? _raw_write_unlock_bh+0x60/0x60                                                                                                                                                                                                                                                                                              
 tfw_tls_connection_recv+0x175/0xaa0 [tempesta_fw]                                                                                                                                                                                                                                                                             
 ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]                                                                                                             
 ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]                                                                                                                 
 ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]                                                                                                                                                                                                                                                                             
 ? tcp_grow_window+0xcb/0x700                                                                                                                                  
 ss_tcp_data_ready+0x141/0x280 [tempesta_fw]                                                                                                                   
 tcp_data_queue+0x1669/0x4cb0                                                                                                                                                                                                                                                                                                  
 ? tcp_data_ready+0x450/0x450                                                                                                                                                                                                                                                                                                  
 ? tcp_try_keep_open+0x2b0/0x2b0                                                                                                                               
 ? kvm_clock_get_cycles+0xd/0x10                                                                                                                               
 ? ktime_get+0x55/0xd0                                                                                                                                                                                                                                                                                                         
 ? tempesta_sock_tcp_rcv+0x8d/0xc0                                                                                                                                                                                                                                                                                             
 tcp_rcv_established+0x6ae/0x1d10                                                                                                                              
 ? tcp_data_queue+0x4cb0/0x4cb0                                                                                                                                
 tcp_v4_do_rcv+0x4fa/0x750                                                                                                                                     
 tcp_v4_rcv+0x2588/0x3530                                                                                                                                      
 ? nf_ct_deliver_cached_events+0xe4/0x410 [nf_conntrack]                                                                                                       
 ? tcp_v4_early_demux+0x790/0x790                                                                                                                                                                                                                                                                                              
 ip_protocol_deliver_rcu+0x6a/0x550                                                                                                                            
 ? nf_confirm+0x340/0x340 [nf_conntrack]                                                                                                                       
 ip_local_deliver_finish+0x1a4/0x250                                                                                                                           
 ip_local_deliver+0x246/0x2a0                                                                                                                                  
 ? ip_local_deliver_finish+0x250/0x250                                                                                                                                                                                                                                                                                         
 ? ip_protocol_deliver_rcu+0x550/0x550                                                                                                                         
 ip_sublist_rcv_finish+0x11f/0x1b0                                                                                                                                                                                                                                                                                             
 ip_sublist_rcv+0x3e5/0x630                                                                                                                                    
 ? ip_rcv_core+0xa70/0xa70                                                                                                                                                                                                                                                                                                     
 ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0                                                                                                                
 ip_list_rcv+0x293/0x3b0                                                                                                                                       
 ? ip_rcv+0x180/0x180                                                                                                                                          
 ? memset+0x20/0x40                                                                                                                                                                                                                                                                                                            
 __netif_receive_skb_list_core+0x61d/0x8b0                                                                                                                                                                                                                                                                                     
 ? page_to_skb+0x6c6/0x9f0 [virtio_net]                                                                                                                                                                                                                                                                                        
 ? __netif_receive_skb_core.constprop.0+0x3050/0x3050                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
 ? gro_normal_one+0x56/0x220                                                                                                                                   
 ? napi_gro_receive+0x23c/0x2a0                                                                                                                                
 ? receive_buf+0xb68/0x57f0 [virtio_net]                                                                                                                       
 netif_receive_skb_list_internal+0x5d7/0xc10                                                                                                                   
 ? __netif_receive_skb_list_core+0x8b0/0x8b0                                                                                                                   
 ? page_to_skb+0x9f0/0x9f0 [virtio_net]                                                                                                                        
 napi_complete_done+0x188/0x5d0                                                                                                                                                                                                                                                                                                
 virtqueue_napi_complete+0x2b/0x80 [virtio_net]                                                                                                                                                                                                                                                                                
 virtnet_poll+0x90c/0xc81 [virtio_net]                                                                                                                                                                                                                                                                                         
 ? receive_buf+0x57f0/0x57f0 [virtio_net]                                                                                                                      
 ? kvm_sched_clock_read+0xd/0x20                                                                                                                                                                                                                                                                                               
 ? sched_clock+0x5/0x10                                                                                                                                        
 ? sched_clock_cpu+0x18/0x160                                                                                                                                                                                                                                                                                                  
 net_rx_action+0x37c/0xbc0                                                                                                                                     
 ? rcu_accelerate_cbs+0x61/0x510                                                                                                                               
 ? napi_complete_done+0x5d0/0x5d0                                                                                                                              
 __do_softirq+0x1a6/0x5ab                                                                                                                                      
 asm_call_irq_on_stack+0xf/0x20                                                                                                                                
 </IRQ>                                                                                                                                                        
 do_softirq_own_stack+0x37/0x40                                                                                                                                
 irq_exit_rcu+0x1ad/0x260                                                                                                                                      
 sysvec_apic_timer_interrupt+0x33/0x80                                                                                                                         
 asm_sysvec_apic_timer_interrupt+0x12/0x20                                                                                                                                                                                                                                                                                     
RIP: 0010:native_safe_halt+0xe/0x10                                                                                                                                                                                                                                                                                            
Code: ff ff ff 4c 89 e7 e8 b1 29 2b fe e9 ed fe ff ff 48 89 ef e8 a4 29 2b fe eb a3 cc cc e9 07 00 00 00 0f 00 2d c4 f7 4d 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d b4 f7 4d 00 f4 c3 cc cc 0f 1f 44 00
RSP: 0018:ffffc90000127e28 EFLAGS: 00000206                                                                                                                                                                                                                                                                                    
RAX: ffffffff835702f0 RBX: ffff888120ab5080 RCX: ffffffff8354ee5d                                                                                                                                                                                                                                                              
RDX: 0000000000000002 RSI: 0000000000000004 RDI: ffff888149732f80                                                                                              
RBP: ffffed1024156a10 R08: 0000000000000001 R09: ffff888149732f83                                                                                              
R10: ffffed10292e65f0 R11: 0000000000000001 R12: 0000000000000002                                                                                              
R13: ffffffff84ce0e20 R14: 0000000000000000 R15: dffffc0000000000                                                                                                                                                                                                                                                              
 ? __sched_text_end+0x7/0x7                                                                                                                                    
 ? rcu_eqs_enter.constprop.0+0x7d/0xa0                                                                                                                         
 default_idle+0xa/0x10                                                                                                                                                                                                                                                                                                         
 default_idle_call+0x74/0x1f0                                                                                                                                  
 do_idle+0x3bd/0x550                                                                                                                                           
 ? arch_cpu_idle_exit+0x40/0x40                                                                                                                                
 cpu_startup_entry+0x19/0x20                                                                                                                                   
 start_secondary+0x22b/0x2d0                                                                                                                                   
 ? set_cpu_sibling_map+0x21e0/0x21e0                                                                                                                           
 ? start_cpu0+0xc/0xc                                                                                                                                                                                                                                                                                                          
 secondary_startup_64_no_verify+0xc2/0xcb                                                                                                                      
---[ end trace 321d13b12076d1c7 ]---                                                                                                                           
[tempesta tls]   send alert level=2 message=50          

s0nx avatar Oct 10 '22 16:10 s0nx

Created the separate issue #1713 for it.

s0nx avatar Oct 10 '22 16:10 s0nx

  • [x] Some strange crash on if-non-match parsing:
[  362.596229] [tempesta fw]   tfw_h2_parse_req_hdr: fin=1, len=15, data="6331593c-3cd1", req=[00000000529c3f7d]
[  362.596236] [tempesta fw]       enter FSM at state 0000000000000000
[  362.596245] [tempesta fw]       parser at Req_HdrIf_None_MatchV: c=0x22("), p_off=0
[  362.596256] [tempesta fw]       enter FSM at state 000000000884cb17
[  362.596263] [tempesta fw]       parser at I_GMT: c=0x20( ), p_off=-81543651
[  362.596278] BUG: kernel NULL pointer dereference, address: 000000000000000f
[  362.596288] #PF: supervisor instruction fetch in kernel mode
[  362.596293] #PF: error_code(0x0010) - not-present page
[  362.596297] PGD 0 P4D 0
[  362.596304] Oops: 0010 [#1] SMP KASAN NOPTI
[  362.596309] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           OE     5.10.35+ #1
[  362.596313] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[  362.596319] RIP: 0010:0xf
[  362.596325] Code: Unable to access opcode bytes at RIP 0xffffffffffffffe5.
[  362.596329] RSP: 0018:ffffc9000024fad0 EFLAGS: 00010282
[  362.596334] RAX: 00000000fffffffe RBX: 0000000000000022 RCX: 0000000000000000
[  362.596337] RDX: 0000000000000020 RSI: 0000000000000000 RDI: ffff88812a302798
[  362.596341] RBP: ffff88812a302778 R08: 0000000000000001 R09: ffff88812a302778
[  362.596344] R10: ffffed10254604f2 R11: 0000000000000001 R12: 0000000000000000
[  362.596348] R13: ffff88812fa3e203 R14: ffffffffc0bd8e5a R15: ffff88812ac7a020
[  362.596352] FS:  0000000000000000(0000) GS:ffff888149780000(0000) knlGS:0000000000000000
[  362.596357] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  362.596360] CR2: ffffffffffffffe5 CR3: 000000017f016004 CR4: 0000000000770ee0
[  362.596374] PKRU: 55555554
[  362.596377] Call Trace:
[  362.596380]  <IRQ>
[  362.596424]  ? tfw_hpack_decode+0x17f6/0x7850 [tempesta_fw]
[  362.596462]  ? tfw_h2_parse_req+0x448/0x8a0 [tempesta_fw]
[  362.596499]  ? ss_skb_process+0x393/0x570 [tempesta_fw]
[  362.596534]  ? tfw_http_parse_req+0x398a0/0x398a0 [tempesta_fw]
[  362.596542]  ? irq_work_queue+0x1a/0x30
[  362.596578]  ? tfw_sock_srv_exit+0x30/0x30 [tempesta_fw]
[  362.596612]  ? tfw_http_req_process+0x1da/0x15a0 [tempesta_fw]
[  362.596646]  ? tfw_http_req_cache_cb+0x17e0/0x17e0 [tempesta_fw]
[  362.596652]  ? dynamic_emit_prefix+0x350/0x350
[  362.596657]  ? alloc_pages_current+0x132/0x210
[  362.596661]  ? __get_free_pages+0x11/0xa0
[  362.596693]  ? __tfw_pool_new+0x25/0x180 [tempesta_fw]
[  362.596725]  ? tfw_http_msg_process_generic+0x883/0x1380 [tempesta_fw]
[  362.596757]  ? tfw_h2_frame_local_resp+0x90/0x90 [tempesta_fw]
[  362.596790]  ? ss_skb_chop_head_tail+0x34e/0x790 [tempesta_fw]
[  362.596824]  ? ss_skb_expand_head_tail+0x2e0/0x2e0 [tempesta_fw]
[  362.596857]  ? ss_skb_split+0x32f/0x480 [tempesta_fw]
[  362.596888]  ? tfw_h2_frame_process+0x3d1/0x12a0 [tempesta_fw]
[  362.596919]  ? tfw_h2_stream_id_close+0x3a0/0x3a0 [tempesta_fw]
[  362.596950]  ? tfw_connection_recv+0x102/0x1e0 [tempesta_fw]
[  362.596983]  ? tfw_tls_connection_recv+0x4d9/0xa30 [tempesta_fw]
[  362.597015]  ? ss_skb_unroll+0x9bd/0xeb0 [tempesta_fw]
[  362.597048]  ? tfw_tls_connection_lost+0x30/0x30 [tempesta_fw]
[  362.597081]  ? ss_tcp_process_data+0x5ef/0xd80 [tempesta_fw]
[  362.597087]  ? deref_stack_reg+0xcf/0x170
[  362.597119]  ? ss_conn_drop_guard_exit+0xd0/0xd0 [tempesta_fw]
[  362.597125]  ? secondary_startup_64_no_verify+0xc2/0xcb
[  362.597158]  ? ss_tcp_data_ready+0x141/0x280 [tempesta_fw]
[  362.597164]  ? tcp_data_queue+0x1669/0x4cb0
[  362.597167]  ? tcp_reset+0x1d0/0x1d0
[  362.597171]  ? tcp_data_ready+0x450/0x450
[  362.597174]  ? tcp_try_keep_open+0x2b0/0x2b0
[  362.597178]  ? kvm_clock_get_cycles+0xd/0x10
[  362.597183]  ? ktime_get+0x55/0xd0
[  362.597187]  ? tempesta_sock_tcp_rcv+0x8d/0xc0
[  362.597191]  ? tcp_rcv_established+0x6ae/0x1d10
[  362.597196]  ? sk_filter_trim_cap+0x2cb/0x710
[  362.597200]  ? tcp_data_queue+0x4cb0/0x4cb0
[  362.597204]  ? tcp_v4_do_rcv+0x4fa/0x750
[  362.597207]  ? tcp_v4_rcv+0x2588/0x3530
[  362.597220]  ? nf_ct_deliver_cached_events+0xe4/0x410 [nf_conntrack]
[  362.597225]  ? __inet_lookup_established+0x27e/0x4d0
[  362.597229]  ? tcp_v4_early_demux+0x790/0x790
[  362.597233]  ? ip_protocol_deliver_rcu+0x6a/0x550
[  362.597242]  ? nf_confirm+0x340/0x340 [nf_conntrack]
[  362.597246]  ? ip_local_deliver_finish+0x1a4/0x250
[  362.597251]  ? ip_local_deliver+0x246/0x2a0
[  362.597254]  ? ip_local_deliver_finish+0x250/0x250
[  362.597258]  ? ip_rcv_finish_core.constprop.0+0x2e9/0x18a0
[  362.597262]  ? ip_protocol_deliver_rcu+0x550/0x550
[  362.597266]  ? ip_rcv+0x15a/0x180
[  362.597270]  ? ip_local_deliver+0x2a0/0x2a0
[  362.597273]  ? secondary_startup_64_no_verify+0xc2/0xcb
[  362.597277]  ? ip_rcv_finish_core.constprop.0+0x18a0/0x18a0
[  362.597282]  ? get_stack_info+0x30/0x90
[  362.597287]  ? __netif_receive_skb_core.constprop.0+0xdd9/0x3050
[  362.597291]  ? deref_stack_reg+0xcf/0x170
[  362.597295]  ? generic_xdp_tx+0x410/0x410
[  362.597300]  ? update_sd_lb_stats.constprop.0+0x375/0x2cc0
[  362.597306]  ? inet_gro_receive+0x9fb/0x11d0
[  362.597310]  ? memset+0x20/0x40
[  362.597313]  ? __netif_receive_skb_list_core+0x2cf/0x8b0
[  362.597320]  ? page_to_skb+0x69b/0x9f0 [virtio_net]
[  362.597324]  ? __netif_receive_skb_core.constprop.0+0x3050/0x3050
[  362.597328]  ? gro_normal_one+0x56/0x220
[  362.597332]  ? napi_gro_receive+0x23c/0x2a0
[  362.597336]  ? receive_buf+0xb68/0x57f0 [virtio_net]
[  362.597340]  ? netif_receive_skb_list_internal+0x5d7/0xc10
[  362.597344]  ? __netif_receive_skb_list_core+0x8b0/0x8b0
[  362.597349]  ? page_to_skb+0x9f0/0x9f0 [virtio_net]
[  362.597383]  ? tfw_apm_prcntl_tmfn+0xcc2/0x1640 [tempesta_fw]
[  362.597388]  ? napi_complete_done+0x188/0x5d0
[  362.597393]  ? virtqueue_napi_complete+0x2b/0x80 [virtio_net]
[  362.597398]  ? virtnet_poll+0x90c/0xc81 [virtio_net]
[  362.597403]  ? receive_buf+0x57f0/0x57f0 [virtio_net]
[  362.597408]  ? net_rx_action+0x37c/0xbc0
[  362.597412]  ? napi_complete_done+0x5d0/0x5d0
[  362.597416]  ? sched_clock_cpu+0x18/0x160
[  362.597421]  ? __do_softirq+0x1a6/0x5ab
[  362.597426]  ? handle_edge_irq+0x35e/0xb60
[  362.597431]  ? asm_call_irq_on_stack+0xf/0x20
[  362.597433]  </IRQ>
[  362.597437]  ? do_softirq_own_stack+0x37/0x40
[  362.597442]  ? irq_exit_rcu+0x1ad/0x260
[  362.597446]  ? common_interrupt+0x71/0x110
[  362.597450]  ? asm_common_interrupt+0x1e/0x40
[  362.597454]  ? __sched_text_end+0x7/0x7
[  362.597458]  ? rcu_eqs_enter.constprop.0+0x7d/0xa0
[  362.597462]  ? native_safe_halt+0xe/0x10
[  362.597466]  ? default_idle+0xa/0x10
[  362.597470]  ? default_idle_call+0x74/0x1f0
[  362.597474]  ? do_idle+0x3bd/0x550
[  362.597478]  ? arch_cpu_idle_exit+0x40/0x40
[  362.597481]  ? cpu_startup_entry+0x19/0x20
[  362.597485]  ? start_secondary+0x22b/0x2d0
[  362.597489]  ? set_cpu_sibling_map+0x21e0/0x21e0
[  362.597493]  ? start_cpu0+0xc/0xc
[  362.597496]  ? secondary_startup_64_no_verify+0xc2/0xcb
[  362.597500] Modules linked in: tempesta_fw(OE) tempesta_db(OE) sha256_ssse3 sha512_ssse3 tempesta_tls(OE) tempesta_lib(OE) xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 ip6tab
le_mangle ip6table_nat ip6table_filter ip6_tables iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_filter bridge stp llc qrtr ns rfkill sunrpc vfat fat kvm_intel
 kvm iTCO_wdt intel_pmc_bxt iTCO_vendor_support irqbypass virtiofs input_leds lpc_ich pcspkr i2c_i801 led_class i2c_smbus virtio_balloon fuse zram ip_tables crct10dif_pclmul crc32_pclmul crc32c_int
el virtio_net ghash_clmulni_intel serio_raw virtio_scsi net_failover failover virtio_console virtio_blk qemu_fw_cfg ipmi_devintf ipmi_msghandler
[  362.597590] CR2: 000000000000000f
[  362.597598] ---[ end trace d3e017451f69f5c6 ]---
[  362.597602] RIP: 0010:0xf
[  362.597607] Code: Unable to access opcode bytes at RIP 0xffffffffffffffe5.
[  362.597611] RSP: 0018:ffffc9000024fad0 EFLAGS: 00010282
[  362.597615] RAX: 00000000fffffffe RBX: 0000000000000022 RCX: 0000000000000000
[  362.597622] RDX: 0000000000000020 RSI: 0000000000000000 RDI: ffff88812a302798
[  362.597625] RBP: ffff88812a302778 R08: 0000000000000001 R09: ffff88812a302778
[  362.597628] R10: ffffed10254604f2 R11: 0000000000000001 R12: 0000000000000000
[  362.597632] R13: ffff88812fa3e203 R14: ffffffffc0bd8e5a R15: ffff88812ac7a020
[  362.597637] FS:  0000000000000000(0000) GS:ffff888149780000(0000) knlGS:0000000000000000
[  362.597640] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  362.597643] CR2: ffffffffffffffe5 CR3: 000000017f016004 CR4: 0000000000770ee0
[  362.597657] PKRU: 55555554
[  362.597661] Kernel panic - not syncing: Fatal exception in interrupt
[  362.597771] Kernel Offset: disabled
[  362.597780] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
gef➤

s0nx avatar Oct 16 '22 22:10 s0nx

Unfortunately, i wasn't able to reproduce the following issues:

s0nx avatar Oct 17 '22 19:10 s0nx