tempesta
tempesta copied to clipboard
Published
20 hours ago •
tempesta-tech
tempesta-tech
Evaluate kmem_cache performance for H2 connection streams.
trafficstars
Motivation
Need to evaluate the performance of kmem_cache versus TfwPool applied to H2 connection streams.
Testing
perf data could be collected while running h2load test as described in #1677
While trying to collect perf data with caching enabled, OOM was encountered (#1728).
Initially, the concern was about tfw_h2_conn_streams_cleanup() performance, because it traverses the whole connection RB tree deleting all the streams in O(n) time.
This function should not be called too often during normal execution, because currently it would only be called when connection with client or backend is terminated.
For example, in this case:
ss_tcp_state_change()
\-> ss_link_error()
\-> ss_conn_drop_guard_exit()
\-> SS_CALL(connection_drop, sk) -> tfw_sock_clnt_drop()
\-> tfw_connection_drop()
\-> TFW_CONN_HOOK_CALL(conn, conn_drop) -> tfw_http_conn_drop()
\-> tfw_h2_conn_streams_cleanup()
When stream is being closed the regular way, it would be erased from the tree and put the closed streams list. This works reasonably fast, as can be observed in perf report.
The tests were performed using h2load tool with caching enabled in Tempesta FW on a VM with 4 CPUs and 4GB RAM. On each run, the cache was warmed up in advance.
- 2 threads, 1000 clients, 10 streams, 1 min run
h2load https://f35tfw.local -t 2 -c 1000 -D 60 -m 10:
Samples: 185K of event 'cycles', Event count (approx.): 198970318227
Children Self Command Shared Object Symbol
+ 69.12% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] ret_from_fork
+ 69.12% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] kthread
+ 69.12% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] smpboot_thread_fn
+ 69.10% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] __do_softirq
+ 69.09% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] run_ksoftirqd
+ 53.79% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] net_rx_action
+ 53.76% 0.03% ksoftirqd/2 [virtio_net] [k] virtnet_poll
+ 51.82% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] napi_gro_receive
+ 51.60% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] gro_normal_one
+ 51.57% 0.05% ksoftirqd/2 [kernel.vmlinux] [k] netif_receive_skb_list_internal
+ 51.49% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] __netif_receive_skb_list_core
+ 51.34% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] ip_list_rcv
+ 51.22% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] ip_sublist_rcv
+ 48.65% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] ip_sublist_rcv_finish
+ 47.85% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] ip_local_deliver_finish
+ 47.84% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] ip_protocol_deliver_rcu
+ 47.80% 0.23% ksoftirqd/2 [kernel.vmlinux] [k] tcp_v4_rcv
+ 46.03% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] tcp_v4_do_rcv
+ 45.12% 0.23% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rcv_established
+ 26.12% 0.21% ksoftirqd/2 [tempesta_fw] [k] ss_tcp_process_data
+ 26.11% 0.00% ksoftirqd/2 [tempesta_fw] [k] ss_tcp_data_ready
+ 25.50% 0.02% ksoftirqd/2 [tempesta_fw] [k] tfw_tls_connection_recv
+ 23.91% 0.06% ksoftirqd/2 [kernel.vmlinux] [k] tcp_data_queue
+ 21.03% 0.01% ksoftirqd/2 [tempesta_fw] [k] tfw_connection_recv
+ 21.02% 0.10% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_process
+ 20.44% 0.99% ksoftirqd/2 [kernel.vmlinux] [k] tcp_write_xmit
+ 18.26% 0.05% ksoftirqd/2 [tempesta_fw] [k] tfw_http_req_process
+ 16.34% 0.01% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_process
+ 16.20% 1.19% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_do_action
+ 14.07% 14.05% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_8_new17529
+ 12.91% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] __tcp_push_pending_frames
+ 11.96% 0.35% ksoftirqd/2 [kernel.vmlinux] [k] __tcp_transmit_skb
+ 11.01% 0.76% ksoftirqd/2 [kernel.vmlinux] [k] tcp_ack
+ 10.88% 0.09% ksoftirqd/2 [kernel.vmlinux] [k] __ip_queue_xmit
+ 7.62% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] tasklet_action_common.constprop.0
+ 7.61% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] tcp_tasklet_func
+ 7.52% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] tcp_tsq_handler
+ 7.39% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] net_tx_action
+ 7.39% 0.56% ksoftirqd/2 [tempesta_fw] [k] ss_tx_action
+ 6.70% 0.34% ksoftirqd/2 [kernel.vmlinux] [k] ip_finish_output2
+ 6.70% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] __kfree_skb
+ 6.63% 0.05% ksoftirqd/2 [tempesta_fw] [k] ss_skb_process
+ 6.35% 0.18% ksoftirqd/2 [kernel.vmlinux] [k] __dev_queue_xmit
+ 6.34% 0.98% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_pages_nodemask
+ 6.15% 0.16% ksoftirqd/2 [kernel.vmlinux] [k] nf_hook_slow
+ 5.81% 4.17% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_data
+ 5.76% 0.05% ksoftirqd/2 [kernel.vmlinux] [k] sch_direct_xmit
+ 5.36% 2.87% ksoftirqd/2 [kernel.vmlinux] [k] get_page_from_freelist
+ 5.03% 0.07% ksoftirqd/2 [kernel.vmlinux] [k] dev_hard_start_xmit
+ 4.96% 0.16% ksoftirqd/2 [virtio_net] [k] start_xmit
+ 4.78% 4.78% ksoftirqd/2 [tempesta_lib] [k] __memcpy_fast
+ 3.99% 0.35% ksoftirqd/2 [tempesta_fw] [k] tfw_tls_encrypt
+ 3.54% 0.08% ksoftirqd/2 [tempesta_tls] [k] ttls_recv
+ 3.43% 1.80% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_skb
+ 3.31% 0.17% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_in
+ 3.07% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] __ip_local_out
+ 2.72% 2.72% ksoftirqd/2 [kernel.vmlinux] [k] __list_del_entry_valid
+ 2.60% 0.04% ksoftirqd/2 [virtio_net] [k] free_old_xmit_skbs
+ 2.49% 0.00% ksoftirqd/2 [tempesta_tls] [k] ttls_handshake_server_step
+ 2.46% 1.88% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock
+ 2.38% 0.00% ksoftirqd/2 [tempesta_tls] [k] ttls_rsa_pkcs1_sign
+ 2.34% 0.04% ksoftirqd/2 [tempesta_tls] [k] ttls_mpi_exp_mod
+ 2.31% 0.00% ksoftirqd/2 [tempesta_tls] [k] ttls_rsa_private
+ 2.29% 0.20% ksoftirqd/2 [tempesta_tls] [k] __mpi_montmul
+ 2.12% 2.11% ksoftirqd/2 [kernel.vmlinux] [k] key_128_enc_update4
+ 2.07% 0.13% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_recv
+ 2.06% 2.06% ksoftirqd/2 [tempesta_tls] [k] __mpi_mul
+ 2.03% 0.63% ksoftirqd/2 [kernel.vmlinux] [k] free_unref_page
+ 2.02% 2.02% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest815034
+ 2.00% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] nf_hook_slow_list
+ 1.96% 1.91% ksoftirqd/2 [ip_tables] [k] ipt_do_table
+ 1.73% 0.09% ksoftirqd/2 [tempesta_tls] [k] ttls_encrypt
+ 1.69% 1.02% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_get_buf_ctx_split
+ 1.67% 0.09% ksoftirqd/2 [kernel.vmlinux] [k] tcp_event_new_data_sent
+ 1.64% 0.64% ksoftirqd/2 [kernel.vmlinux] [k] pg_skb_alloc
+ 1.55% 1.54% ksoftirqd/2 [nf_conntrack] [k] __nf_conntrack_find_get
+ 1.53% 0.19% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_h2_write
+ 1.50% 0.30% ksoftirqd/2 [tempesta_fw] [k] tfw_http_msg_expand_data
+ 1.48% 1.47% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init_avx_gen4
+ 1.42% 0.33% ksoftirqd/2 [virtio_net] [k] receive_buf
+ 1.26% 0.04% ksoftirqd/2 [kernel.vmlinux] [k] sk_stream_alloc_skb
+ 1.24% 1.24% ksoftirqd/2 [kernel.vmlinux] [k] rb_erase
+ 1.20% 0.32% ksoftirqd/2 [kernel.vmlinux] [k] crypto_aead_encrypt
+ 1.17% 1.17% ksoftirqd/2 [kernel.vmlinux] [k] __raw_callee_save___pv_queued_spin_unlock
+ 1.15% 0.04% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_set_status.constprop.0
+ 1.12% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] kvm_clock_get_cycles
+ 1.12% 1.12% ksoftirqd/2 [kernel.vmlinux] [k] pvclock_clocksource_read
+ 1.11% 0.46% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_tcp_packet
+ 1.10% 0.06% ksoftirqd/2 [kernel.vmlinux] [k] tcp_fragment
+ 1.08% 1.08% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_717529
+ 1.07% 0.17% ksoftirqd/2 [kernel.vmlinux] [k] __mod_timer
+ 1.07% 0.26% ksoftirqd/2 [virtio_net] [k] page_to_skb
+ 1.05% 1.05% ksoftirqd/2 [kernel.vmlinux] [k] page_frag_free
+ 1.04% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] sk_reset_timer
+ 1.02% 0.24% ksoftirqd/2 [kernel.vmlinux] [k] gcmaes_crypt_by_sg
+ 0.98% 0.01% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_parse_req
+ 0.97% 0.17% ksoftirqd/2 [tempesta_fw] [k] tfw_hpack_decode
+ 0.95% 0.04% ksoftirqd/2 [kernel.vmlinux] [k] free_pcppages_bulk
+ 0.95% 0.18% ksoftirqd/2 [tempesta_tls] [k] ttls_decrypt
+ 0.95% 0.02% ksoftirqd/2 [tempesta_fw] [k] ss_skb_expand_head_tail
+ 0.92% 0.59% ksoftirqd/2 [tempesta_fw] [k] __new_pgfrag
0.87% 0.43% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_build_resp_hdr
0.85% 0.41% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_add_split
+ 0.85% 0.85% ksoftirqd/2 [kernel.vmlinux] [k] rb_first
+ 0.84% 0.81% ksoftirqd/2 [kernel.vmlinux] [k] __inet_lookup_established
+ 0.82% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_add_outbuf
+ 0.81% 0.04% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_stream_id_close
+ 0.79% 0.02% ksoftirqd/2 [tempesta_fw] [k] ss_skb_split
+ 0.78% 0.07% ksoftirqd/2 [kernel.vmlinux] [k] tcp_schedule_loss_probe.part.0
+ 0.78% 0.77% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rbtree_insert
+ 0.77% 0.11% ksoftirqd/2 [kernel.vmlinux] [k] ktime_get
+ 0.76% 0.04% ksoftirqd/2 [kernel.vmlinux] [k] ip_local_deliver
+ 0.75% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] consume_skb
0.74% 0.43% ksoftirqd/2 [kernel.vmlinux] [k] alloc_pages_current
+ 0.68% 0.68% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_817529
+ 0.68% 0.04% ksoftirqd/2 [kernel.vmlinux] [k] ip_output
+ 0.68% 0.01% ksoftirqd/2 [tempesta_fw] [k] tfw_http_req_cache_cb
+ 0.67% 0.64% ksoftirqd/2 [kernel.vmlinux] [k] skb_split
0.67% 0.30% ksoftirqd/2 [kernel.vmlinux] [k] detach_buf_split
+ 0.65% 0.65% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_117529
0.63% 0.38% ksoftirqd/2 [kernel.vmlinux] [k] kfree_skbmem
0.63% 0.13% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_head_state
+ 0.61% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] __napi_alloc_skb
+ 0.61% 0.61% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_617529
+ 0.59% 0.59% ksoftirqd/2 [kernel.vmlinux] [k] __pv_queued_spin_lock_slowpath
+ 0.57% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] scatterwalk_map_and_copy
0.57% 0.18% ksoftirqd/2 [kernel.vmlinux] [k] crypto_aead_decrypt
0.56% 0.23% ksoftirqd/2 [kernel.vmlinux] [k] kmem_cache_free
0.55% 0.08% ksoftirqd/2 [kernel.vmlinux] [k] ktime_get_real_ts64
+ 0.52% 0.52% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_find_stream
0.52% 0.04% ksoftirqd/2 [kernel.vmlinux] [k] ip_rcv_finish_core.constprop.0
0.52% 0.06% ksoftirqd/2 [kernel.vmlinux] [k] skb_to_sgvec
+ 0.52% 0.52% ksoftirqd/2 [kernel.vmlinux] [k] _count_set_17530
0.50% 0.49% ksoftirqd/2 [kernel.vmlinux] [k] rb_insert_color
0.49% 0.36% ksoftirqd/2 [kernel.vmlinux] [k] sg_init_table
- 2 threads, 1000 clients, 100 streams, 1 min run
h2load https://f35tfw.local -t 2 -c 1000 -D 60 -m 100:
Samples: 262K of event 'cycles', Event count (approx.): 199480018345
Children Self Command Shared Object Symbol
+ 63.04% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] ret_from_fork ◆
+ 63.04% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] kthread ▒
+ 63.04% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] smpboot_thread_fn ▒
+ 63.02% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] __do_softirq ▒
+ 62.99% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] run_ksoftirqd ▒
+ 53.77% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] net_rx_action ▒
+ 52.75% 0.02% ksoftirqd/2 [virtio_net] [k] virtnet_poll ▒
+ 52.35% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] netif_receive_skb_list_internal ▒
+ 52.28% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] __netif_receive_skb_list_core ▒
+ 51.92% 0.13% ksoftirqd/2 [kernel.vmlinux] [k] __netif_receive_skb_core.constprop.0 ▒
+ 51.89% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] gro_normal_one ▒
+ 51.51% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] napi_gro_receive ▒
+ 49.49% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] ip_local_deliver_finish ▒
+ 49.48% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] ip_protocol_deliver_rcu ▒
+ 49.46% 0.11% ksoftirqd/2 [kernel.vmlinux] [k] tcp_v4_rcv ▒
+ 47.89% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] tcp_v4_do_rcv ▒
+ 47.72% 0.09% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rcv_established ▒
+ 26.53% 0.04% ksoftirqd/2 [tempesta_fw] [k] ss_tcp_process_data ▒
+ 26.52% 0.00% ksoftirqd/2 [tempesta_fw] [k] ss_tcp_data_ready ▒
+ 26.34% 0.02% ksoftirqd/2 [tempesta_fw] [k] tfw_tls_connection_recv ▒
+ 23.25% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] tcp_data_queue ▒
+ 22.82% 0.00% ksoftirqd/2 [tempesta_fw] [k] tfw_connection_recv ▒
+ 22.80% 0.05% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_process ▒
+ 20.63% 0.06% ksoftirqd/2 [tempesta_fw] [k] tfw_http_req_process ▒
+ 18.90% 0.02% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_process ▒
+ 18.75% 1.28% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_do_action ▒
+ 17.09% 0.55% ksoftirqd/2 [kernel.vmlinux] [k] tcp_write_xmit ▒
+ 15.00% 14.97% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_8_new17529 ▒
+ 13.95% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] __tcp_push_pending_frames ▒
+ 10.80% 0.53% ksoftirqd/2 [kernel.vmlinux] [k] tcp_ack ▒
+ 9.42% 0.27% ksoftirqd/2 [kernel.vmlinux] [k] __tcp_transmit_skb ▒
+ 8.57% 0.06% ksoftirqd/2 [kernel.vmlinux] [k] __ip_queue_xmit ▒
+ 7.36% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] __kfree_skb ▒
+ 6.82% 1.01% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_pages_nodemask ▒
+ 6.70% 4.32% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_data ▒
+ 5.90% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] net_tx_action ▒
+ 5.89% 0.85% ksoftirqd/2 [tempesta_fw] [k] ss_tx_action ▒
+ 5.87% 2.82% ksoftirqd/2 [kernel.vmlinux] [k] get_page_from_freelist ▒
+ 5.86% 5.85% ksoftirqd/2 [tempesta_lib] [k] __memcpy_fast ▒
+ 5.53% 0.26% ksoftirqd/2 [kernel.vmlinux] [k] ip_finish_output2 ▒
+ 5.26% 0.15% ksoftirqd/2 [kernel.vmlinux] [k] __dev_queue_xmit ▒
+ 4.75% 0.03% ksoftirqd/2 [kernel.vmlinux] [k] sch_direct_xmit ▒
+ 4.63% 0.04% ksoftirqd/2 [tempesta_fw] [k] ss_skb_process ▒
+ 4.07% 0.05% ksoftirqd/2 [kernel.vmlinux] [k] dev_hard_start_xmit ▒
+ 4.06% 0.11% ksoftirqd/2 [kernel.vmlinux] [k] nf_hook_slow ▒
+ 4.02% 0.15% ksoftirqd/2 [virtio_net] [k] start_xmit ▒
+ 3.84% 3.83% ksoftirqd/2 [kernel.vmlinux] [k] key_128_enc_update4 ▒
+ 3.60% 3.60% ksoftirqd/2 [kernel.vmlinux] [k] __list_del_entry_valid ▒
+ 3.17% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] tasklet_action_common.constprop.0 ▒
+ 3.16% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] tcp_tasklet_func ▒
+ 3.16% 0.22% ksoftirqd/2 [tempesta_fw] [k] tfw_tls_encrypt ▒
+ 3.13% 0.00% ksoftirqd/2 [kernel.vmlinux] [k] tcp_tsq_handler ▒
+ 3.05% 1.58% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_skb ▒
+ 2.97% 1.64% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock ▒
+ 2.96% 0.62% ksoftirqd/2 [kernel.vmlinux] [k] free_unref_page ▒
+ 2.92% 0.03% ksoftirqd/2 [tempesta_tls] [k] ttls_recv ▒
+ 2.50% 0.00% ksoftirqd/2 [tempesta_tls] [k] ttls_handshake_server_step ▒
+ 2.37% 0.00% ksoftirqd/2 [tempesta_tls] [k] ttls_rsa_pkcs1_sign ▒
+ 2.33% 0.03% ksoftirqd/2 [tempesta_tls] [k] ttls_mpi_exp_mod ▒
+ 2.32% 0.00% ksoftirqd/2 [tempesta_tls] [k] ttls_rsa_private ▒
+ 2.29% 0.20% ksoftirqd/2 [tempesta_tls] [k] __mpi_montmul ▒
+ 2.23% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] ret_from_fork ▒
+ 2.23% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] kthread ▒
+ 2.23% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] worker_thread ▒
+ 2.22% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] process_one_work ▒
+ 2.22% 0.00% kworker/2:2-eve [nf_conntrack] [k] gc_worker ▒
+ 2.22% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] do_softirq_own_stack ▒
+ 2.22% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] asm_call_irq_on_stack ▒
+ 2.22% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __do_softirq ▒
+ 2.21% 0.01% kworker/2:2-eve [kernel.vmlinux] [k] __local_bh_enable_ip ▒
+ 2.21% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] do_softirq ▒
+ 2.20% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] __ip_local_out ▒
+ 2.06% 2.06% ksoftirqd/2 [tempesta_tls] [k] __mpi_mul ▒
+ 2.03% 2.03% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest815034 ▒
+ 2.01% 0.13% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_in ▒
+ 1.96% 0.03% ksoftirqd/2 [virtio_net] [k] free_old_xmit_skbs ▒
+ 1.85% 0.08% ksoftirqd/2 [kernel.vmlinux] [k] free_pcppages_bulk ▒
+ 1.81% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] net_rx_action ▒
+ 1.79% 0.00% kworker/2:2-eve [virtio_net] [k] virtnet_poll ▒
+ 1.76% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] netif_receive_skb_list_internal ▒
+ 1.76% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __netif_receive_skb_list_core ▒
+ 1.76% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] gro_normal_one ▒
+ 1.75% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] __netif_receive_skb_core.constprop.0 ▒
+ 1.75% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] napi_gro_receive ▒
+ 1.73% 0.07% ksoftirqd/2 [kernel.vmlinux] [k] tcp_event_new_data_sent ▒
+ 1.67% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] ip_local_deliver_finish ▒
+ 1.67% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] ip_protocol_deliver_rcu ▒
+ 1.67% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] tcp_v4_rcv ▒
+ 1.62% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] tcp_v4_do_rcv ▒
+ 1.62% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] tcp_rcv_established ▒
+ 1.62% 0.22% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_h2_write ▒
+ 1.56% 0.33% ksoftirqd/2 [tempesta_fw] [k] tfw_http_msg_expand_data ▒
+ 1.56% 1.55% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init_avx_gen4 ▒
+ 1.51% 1.48% ksoftirqd/2 [ip_tables] [k] ipt_do_table ▒
+ 1.50% 0.01% swapper [kernel.vmlinux] [k] do_idle ▒
+ 1.50% 0.00% swapper [kernel.vmlinux] [k] secondary_startup_64_no_verify ▒
+ 1.50% 0.00% swapper [kernel.vmlinux] [k] cpu_startup_entry ▒
+ 1.48% 0.57% ksoftirqd/2 [kernel.vmlinux] [k] pg_skb_alloc ▒
+ 1.43% 0.01% swapper [kernel.vmlinux] [k] default_idle_call ▒
+ 1.41% 0.01% swapper [kernel.vmlinux] [k] default_idle ▒
+ 1.40% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] ip_rcv ▒
+ 1.40% 0.00% kworker/2:2-eve [kernel.vmlinux] [k] nf_conntrack_destroy ▒
+ 1.40% 0.06% swapper [kernel.vmlinux] [k] native_safe_halt ▒
+ 1.38% 0.07% ksoftirqd/2 [kernel.vmlinux] [k] tcp_fragment ▒
+ 1.34% 1.33% ksoftirqd/2 [kernel.vmlinux] [k] __pv_queued_spin_lock_slowpath ▒
+ 1.31% 0.81% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_get_buf_ctx_split ▒
+ 1.24% 0.08% ksoftirqd/2 [tempesta_tls] [k] ttls_encrypt ▒
+ 1.21% 0.05% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_set_status.constprop.0 ▒
+ 1.14% 0.00% swapper [kernel.vmlinux] [k] asm_common_interrupt ▒
+ 1.14% 0.00% swapper [kernel.vmlinux] [k] common_interrupt ▒
+ 1.13% 0.00% swapper [kernel.vmlinux] [k] asm_call_irq_on_stack ▒
+ 1.11% 1.11% ksoftirqd/2 [kernel.vmlinux] [k] rb_erase ▒
+ 1.09% 0.02% ksoftirqd/2 [tempesta_fw] [k] ss_skb_split ▒
+ 1.02% 0.79% ksoftirqd/2 [kernel.vmlinux] [k] __free_one_page ▒
+ 1.00% 0.00% swapper [kernel.vmlinux] [k] irq_exit_rcu ▒
+ 1.00% 1.00% ksoftirqd/2 [kernel.vmlinux] [k] __raw_callee_save___pv_queued_spin_unlock ▒
+ 1.00% 0.10% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_recv ▒
+ 0.98% 0.00% swapper [kernel.vmlinux] [k] do_softirq_own_stack ▒
+ 0.98% 0.00% swapper [kernel.vmlinux] [k] __do_softirq ▒
+ 0.97% 0.01% ksoftirqd/2 [kernel.vmlinux] [k] kvm_clock_get_cycles ▒
+ 0.97% 0.97% ksoftirqd/2 [kernel.vmlinux] [k] pvclock_clocksource_read ▒
+ 0.95% 0.02% ksoftirqd/2 [tempesta_fw] [k] ss_skb_expand_head_tail ▒
+ 0.95% 0.95% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_817529 ▒
+ 0.93% 0.76% ksoftirqd/2 [tempesta_fw] [k] __new_pgfrag ▒
+ 0.93% 0.55% ksoftirqd/2 [kernel.vmlinux] [k] alloc_pages_current ▒
+ 0.93% 0.93% ksoftirqd/2 [kernel.vmlinux] [k] page_frag_free ▒
+ 0.92% 0.16% ksoftirqd/2 [kernel.vmlinux] [k] crypto_aead_encrypt ▒
0.92% 0.44% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_build_resp_hdr ▒
+ 0.91% 0.05% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_stream_id_close ▒
+ 0.91% 0.91% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rbtree_insert ▒
+ 0.91% 0.02% ksoftirqd/2 [kernel.vmlinux] [k] sk_stream_alloc_skb
The profiles seems cumulative, so it's hard to say what is the bottleneck. @s0nx could you please show non-cumulative profile, like shown by perf top?
Output of perf report --call-graph=none --no-children
- 2 threads, 1000 clients, 10 streams, 1 min run
h2load https://f35tfw.local -t 2 -c 1000 -D 60 -m 10:
14.05% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_8_new17529
4.78% ksoftirqd/2 [tempesta_lib] [k] __memcpy_fast
4.17% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_data
2.87% ksoftirqd/2 [kernel.vmlinux] [k] get_page_from_freelist
2.72% ksoftirqd/2 [kernel.vmlinux] [k] __list_del_entry_valid
2.11% ksoftirqd/2 [kernel.vmlinux] [k] key_128_enc_update4
2.06% ksoftirqd/2 [tempesta_tls] [k] __mpi_mul
2.02% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest815034
1.91% ksoftirqd/2 [ip_tables] [k] ipt_do_table
1.88% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock
1.80% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_skb
1.54% ksoftirqd/2 [nf_conntrack] [k] __nf_conntrack_find_get
1.47% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init_avx_gen4
1.24% ksoftirqd/2 [kernel.vmlinux] [k] rb_erase
1.19% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_do_action
1.17% ksoftirqd/2 [kernel.vmlinux] [k] __raw_callee_save___pv_queued_spin_unlock
1.12% ksoftirqd/2 [kernel.vmlinux] [k] pvclock_clocksource_read
1.08% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_717529
1.05% ksoftirqd/2 [kernel.vmlinux] [k] page_frag_free
1.02% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_get_buf_ctx_split
0.99% ksoftirqd/2 [kernel.vmlinux] [k] tcp_write_xmit
0.98% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.85% ksoftirqd/2 [kernel.vmlinux] [k] rb_first
0.81% ksoftirqd/2 [kernel.vmlinux] [k] __inet_lookup_established
0.77% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rbtree_insert
0.76% ksoftirqd/2 [kernel.vmlinux] [k] tcp_ack
0.68% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_817529
0.65% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_117529
0.64% ksoftirqd/2 [kernel.vmlinux] [k] pg_skb_alloc
0.64% ksoftirqd/2 [kernel.vmlinux] [k] skb_split
0.63% ksoftirqd/2 [kernel.vmlinux] [k] free_unref_page
0.61% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_617529
0.59% ksoftirqd/2 [kernel.vmlinux] [k] __pv_queued_spin_lock_slowpath
0.59% ksoftirqd/2 [tempesta_fw] [k] __new_pgfrag
0.56% ksoftirqd/2 [tempesta_fw] [k] ss_tx_action
0.52% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_find_stream
0.52% ksoftirqd/2 [kernel.vmlinux] [k] _count_set_17530
0.49% ksoftirqd/2 [kernel.vmlinux] [k] rb_insert_color
0.49% ksoftirqd/2 [tempesta_fw] [k] tfw_hpack_find_index
0.47% ksoftirqd/2 [kernel.vmlinux] [k] crypto_stats_aead_encrypt
0.46% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_tcp_packet
0.46% ksoftirqd/2 [tempesta_lib] [k] __bzero_fast
0.46% ksoftirqd/2 [kernel.vmlinux] [k] __skb_to_sgvec
0.46% ksoftirqd/2 [kernel.vmlinux] [k] slab_free_freelist_hook
0.46% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init
0.45% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_kick_prepare
0.45% ksoftirqd/2 [kernel.vmlinux] [k] scatterwalk_copychunks
0.43% ksoftirqd/2 [kernel.vmlinux] [k] alloc_pages_current
0.43% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_build_resp_hdr
0.43% ksoftirqd/2 [kernel.vmlinux] [k] tcp_wfree
0.42% ksoftirqd/2 [kernel.vmlinux] [k] __list_add_valid
0.42% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_417529
0.41% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_add_split
0.39% ksoftirqd/2 [kernel.vmlinux] [k] iowrite16
0.38% ksoftirqd/2 [kernel.vmlinux] [k] kfree_skbmem
0.37% ksoftirqd/2 [kernel.vmlinux] [k] crypto_stats_get
0.36% ksoftirqd/2 [kernel.vmlinux] [k] sg_init_table
0.36% ksoftirqd/2 [kernel.vmlinux] [k] _eight_cipher_left17529
0.35% ksoftirqd/2 [tempesta_fw] [k] tfw_tls_encrypt
0.35% ksoftirqd/2 [kernel.vmlinux] [k] __tcp_transmit_skb
0.34% ksoftirqd/2 [kernel.vmlinux] [k] skb_clone
0.34% ksoftirqd/2 [kernel.vmlinux] [k] ip_finish_output2
0.34% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_8_new2497
0.33% ksoftirqd/2 [virtio_net] [k] receive_buf
0.33% ksoftirqd/2 [kernel.vmlinux] [k] __free_one_page
0.32% ksoftirqd/2 [nf_conntrack] [k] hash_conntrack_raw
0.32% ksoftirqd/2 [kernel.vmlinux] [k] crypto_aead_encrypt
0.31% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_done15034
0.30% ksoftirqd/2 [tempesta_fw] [k] tfw_http_msg_expand_data
0.30% ksoftirqd/2 [kernel.vmlinux] [k] detach_buf_split
0.30% ksoftirqd/2 [nf_conntrack] [k] nf_ct_seq_offset
0.30% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_317529
0.29% ksoftirqd/2 [kernel.vmlinux] [k] prep_new_page
0.28% ksoftirqd/2 [kernel.vmlinux] [k] skb_entail
0.27% ksoftirqd/2 [tempesta_tls] [k] ttls_aead_req_alloc
0.26% ksoftirqd/2 [virtio_net] [k] page_to_skb
0.25% ksoftirqd/2 [kernel.vmlinux] [k] policy_nodemask
0.24% ksoftirqd/2 [kernel.vmlinux] [k] gcmaes_crypt_by_sg
0.23% ksoftirqd/2 [kernel.vmlinux] [k] kmem_cache_free
0.23% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rcv_established
0.23% ksoftirqd/2 [kernel.vmlinux] [k] tcp_v4_rcv
0.22% ksoftirqd/2 [kernel.vmlinux] [k] __kmalloc
0.22% ksoftirqd/2 [nf_nat] [k] nf_nat_inet_fn
0.22% ksoftirqd/2 [tempesta_fw] [k] tfw_http_msg_hdr_close
0.21% ksoftirqd/2 [tempesta_fw] [k] __tfw_http_msg_alloc
0.21% ksoftirqd/2 [kernel.vmlinux] [k] fib_table_lookup
0.21% ksoftirqd/2 [tempesta_fw] [k] ss_tcp_process_data
0.20% ksoftirqd/2 [kernel.vmlinux] [k] ip_send_check
0.20% ksoftirqd/2 [kernel.vmlinux] [k] free_unref_page_commit
0.20% ksoftirqd/2 [tempesta_tls] [k] __mpi_montmul
0.19% ksoftirqd/2 [kernel.vmlinux] [k] crypto_stats_aead_decrypt
0.19% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.19% ksoftirqd/2 [kernel.vmlinux] [k] __x86_indirect_thunk_rax
0.19% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_h2_write
0.19% ksoftirqd/2 [kernel.vmlinux] [k] __local_bh_enable_ip
0.19% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest82
0.18% ksoftirqd/2 [kernel.vmlinux] [k] __cgroup_bpf_run_filter_skb
0.18% ksoftirqd/2 [kernel.vmlinux] [k] crypto_aead_decrypt
0.18% ksoftirqd/2 [kernel.vmlinux] [k] __dev_queue_xmit
0.18% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_enable_cb_delayed
0.18% ksoftirqd/2 [tempesta_tls] [k] ttls_decrypt
0.17% ksoftirqd/2 [kernel.vmlinux] [k] cryptd_aead_queued
0.17% ksoftirqd/2 [kernel.vmlinux] [k] __mod_timer
0.17% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_in
0.17% ksoftirqd/2 [kernel.vmlinux] [k] kfree
0.17% ksoftirqd/2 [tempesta_fw] [k] tfw_hpack_decode
0.17% ksoftirqd/2 [kernel.vmlinux] [k] memset_erms
0.17% ksoftirqd/2 [kernel.vmlinux] [k] __zone_watermark_ok
0.17% ksoftirqd/2 [kernel.vmlinux] [k] _no_extra_mask_2_17530
0.16% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init_avx_gen2
0.16% ksoftirqd/2 [kernel.vmlinux] [k] nf_hook_slow
0.16% ksoftirqd/2 [virtio_net] [k] start_xmit
0.15% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock_bh
0.15% ksoftirqd/2 [kernel.vmlinux] [k] rb_next
0.15% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_get_buf
0.14% ksoftirqd/2 [kernel.vmlinux] [k] detach_if_pending
0.14% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.13% ksoftirqd/2 [kernel.vmlinux] [k] memcpy_erms
0.13% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_recv
0.13% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_head_state
0.13% ksoftirqd/2 [tempesta_fw] [k] tfw_http_conn_msg_alloc
0.13% ksoftirqd/2 [kernel.vmlinux] [k] sg_next
0.12% ksoftirqd/2 [kernel.vmlinux] [k] __inc_numa_state
0.12% ksoftirqd/2 [tempesta_tls] [k] ttls_xfrm_ready
0.12% ksoftirqd/2 [tempesta_fw] [k] tfw_vhost_lookup_default
0.11% ksoftirqd/2 [kernel.vmlinux] [k] tcp_validate_incoming
0.11% ksoftirqd/2 [kernel.vmlinux] [k] ktime_get
0.11% ksoftirqd/2 [kernel.vmlinux] [k] ___slab_alloc
0.11% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_enc_update_avx_gen4
0.10% ksoftirqd/2 [kernel.vmlinux] [k] _zero_cipher_left17529
0.10% ksoftirqd/2 [kernel.vmlinux] [k] enqueue_timer
0.10% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_process
0.10% ksoftirqd/2 [kernel.vmlinux] [k] tcp_established_options
0.10% ksoftirqd/2 [kernel.vmlinux] [k] __slab_free
0.09% ksoftirqd/2 [kernel.vmlinux] [k] vring_map_one_sg
0.09% ksoftirqd/2 [kernel.vmlinux] [k] tcp_event_new_data_sent
0.09% ksoftirqd/2 [tempesta_tls] [k] ttls_encrypt
0.09% ksoftirqd/2 [kernel.vmlinux] [k] __netif_receive_skb_core.constprop.0
0.09% swapper [kernel.vmlinux] [k] _encrypt_by_8_new17529
0.09% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_217529
0.09% ksoftirqd/2 [kernel.vmlinux] [k] dev_gro_receive
0.09% ksoftirqd/2 [kernel.vmlinux] [k] tcp_small_queue_check.isra.0
0.09% ksoftirqd/2 [tempesta_fw] [k] tfw_wq_pop_ticket
0.09% ksoftirqd/2 [virtio_net] [k] try_fill_recv
0.09% ksoftirqd/2 [kernel.vmlinux] [k] ip_rcv_core
0.09% ksoftirqd/2 [kernel.vmlinux] [k] __ip_queue_xmit
0.09% ksoftirqd/2 [tempesta_fw] [k] tfw_capolicy_match
0.08% ksoftirqd/2 [tempesta_fw] [k] tfw_http_tbl_action
0.08% ksoftirqd/2 [kernel.vmlinux] [k] __ip_finish_output
0.08% ksoftirqd/2 [kernel.vmlinux] [k] skb_put
0.08% ksoftirqd/2 [kernel.vmlinux] [k] __put_page
0.08% ksoftirqd/2 [tempesta_tls] [k] ttls_recv
0.08% ksoftirqd/2 [kernel.vmlinux] [k] ktime_get_real_ts64
- 2 threads, 1000 clients, 100 streams, 1 min run
h2load https://f35tfw.local -t 2 -c 1000 -D 60 -m 100:
14.97% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_8_new17529
5.85% ksoftirqd/2 [tempesta_lib] [k] __memcpy_fast
4.32% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_data
3.83% ksoftirqd/2 [kernel.vmlinux] [k] key_128_enc_update4
3.60% ksoftirqd/2 [kernel.vmlinux] [k] __list_del_entry_valid
2.82% ksoftirqd/2 [kernel.vmlinux] [k] get_page_from_freelist
2.06% ksoftirqd/2 [tempesta_tls] [k] __mpi_mul
2.03% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest815034
1.64% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock
1.58% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_skb
1.55% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init_avx_gen4
1.48% ksoftirqd/2 [ip_tables] [k] ipt_do_table
1.33% ksoftirqd/2 [kernel.vmlinux] [k] __pv_queued_spin_lock_slowpath
1.28% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_do_action
1.11% ksoftirqd/2 [kernel.vmlinux] [k] rb_erase
1.01% ksoftirqd/2 [kernel.vmlinux] [k] __alloc_pages_nodemask
1.00% ksoftirqd/2 [kernel.vmlinux] [k] __raw_callee_save___pv_queued_spin_unlock
0.97% ksoftirqd/2 [kernel.vmlinux] [k] pvclock_clocksource_read
0.95% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_817529
0.93% ksoftirqd/2 [kernel.vmlinux] [k] page_frag_free
0.91% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rbtree_insert
0.88% ksoftirqd/2 [nf_conntrack] [k] __nf_conntrack_find_get
0.87% ksoftirqd/2 [kernel.vmlinux] [k] skb_split
0.85% ksoftirqd/2 [tempesta_fw] [k] ss_tx_action
0.82% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_717529
0.81% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_get_buf_ctx_split
0.79% ksoftirqd/2 [kernel.vmlinux] [k] __free_one_page
0.76% ksoftirqd/2 [tempesta_fw] [k] __new_pgfrag
0.63% kworker/2:2-eve [kernel.vmlinux] [k] _encrypt_by_8_new17529
0.62% ksoftirqd/2 [kernel.vmlinux] [k] free_unref_page
0.62% ksoftirqd/2 [kernel.vmlinux] [k] rb_first
0.57% ksoftirqd/2 [kernel.vmlinux] [k] pg_skb_alloc
0.56% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_617529
0.55% ksoftirqd/2 [kernel.vmlinux] [k] alloc_pages_current
0.55% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_317529
0.55% ksoftirqd/2 [kernel.vmlinux] [k] tcp_write_xmit
0.53% ksoftirqd/2 [kernel.vmlinux] [k] _count_set_17530
0.53% ksoftirqd/2 [kernel.vmlinux] [k] tcp_ack
0.51% ksoftirqd/2 [kernel.vmlinux] [k] rb_insert_color
0.44% ksoftirqd/2 [kernel.vmlinux] [k] crypto_stats_aead_encrypt
0.44% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_build_resp_hdr
0.40% ksoftirqd/2 [kernel.vmlinux] [k] __list_add_valid
0.39% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_kick_prepare
0.39% ksoftirqd/2 [kernel.vmlinux] [k] __skb_to_sgvec
0.39% ksoftirqd/2 [kernel.vmlinux] [k] kfree_skbmem
0.39% swapper [kernel.vmlinux] [k] irq_entries_start
0.38% ksoftirqd/2 [kernel.vmlinux] [k] _eight_cipher_left17529
0.37% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_add_split
0.36% ksoftirqd/2 [kernel.vmlinux] [k] skb_clone
0.35% ksoftirqd/2 [kernel.vmlinux] [k] iowrite16
0.34% ksoftirqd/2 [kernel.vmlinux] [k] scatterwalk_copychunks
0.33% ksoftirqd/2 [tempesta_fw] [k] tfw_http_msg_expand_data
0.31% ksoftirqd/2 [kernel.vmlinux] [k] __inet_lookup_established
0.30% ksoftirqd/2 [kernel.vmlinux] [k] policy_nodemask
0.30% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_done15034
0.30% ksoftirqd/2 [kernel.vmlinux] [k] crypto_stats_get
0.30% ksoftirqd/2 [virtio_net] [k] receive_buf
0.29% ksoftirqd/2 [kernel.vmlinux] [k] skb_entail
0.29% ksoftirqd/2 [kernel.vmlinux] [k] aesni_gcm_init
0.29% ksoftirqd/2 [kernel.vmlinux] [k] prep_new_page
0.29% ksoftirqd/2 [kernel.vmlinux] [k] sg_init_table
0.28% swapper [kernel.vmlinux] [k] _encrypt_by_8_new17529
0.28% ksoftirqd/2 [tempesta_lib] [k] __bzero_fast
0.27% ksoftirqd/2 [kernel.vmlinux] [k] __tcp_transmit_skb
0.27% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_tcp_packet
0.26% ksoftirqd/2 [kernel.vmlinux] [k] ip_finish_output2
0.24% ksoftirqd/2 [kernel.vmlinux] [k] gcmaes_crypt_by_sg
0.24% ksoftirqd/2 [kernel.vmlinux] [k] slab_free_freelist_hook
0.24% ksoftirqd/2 [kernel.vmlinux] [k] tcp_wfree
0.24% ksoftirqd/2 [nf_conntrack] [k] hash_conntrack_raw
0.24% ksoftirqd/2 [kernel.vmlinux] [k] detach_buf_split
0.22% ksoftirqd/2 [tempesta_fw] [k] tfw_tls_encrypt
0.22% ksoftirqd/2 [tempesta_fw] [k] tfw_cache_h2_write
0.22% ksoftirqd/2 [tempesta_fw] [k] __tfw_http_msg_alloc
0.22% ksoftirqd/2 [tempesta_fw] [k] tfw_http_msg_hdr_close
0.21% ksoftirqd/2 [kernel.vmlinux] [k] kmem_cache_free
0.21% ksoftirqd/2 [kernel.vmlinux] [k] free_unref_page_commit
0.20% kworker/2:2-eve [tempesta_lib] [k] __memcpy_fast
0.20% ksoftirqd/2 [kernel.vmlinux] [k] _no_extra_mask_2_17530
0.20% ksoftirqd/2 [tempesta_tls] [k] __mpi_montmul
0.18% swapper [kernel.vmlinux] [k] __pv_queued_spin_lock_slowpath
0.18% ksoftirqd/2 [tempesta_fw] [k] tfw_hpack_find_index
0.17% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_417529
0.17% ksoftirqd/2 [kernel.vmlinux] [k] __kmalloc
0.17% ksoftirqd/2 [kernel.vmlinux] [k] __cgroup_bpf_run_filter_skb
0.17% kworker/2:2-eve [kernel.vmlinux] [k] key_128_enc_update4
0.16% ksoftirqd/2 [kernel.vmlinux] [k] __local_bh_enable_ip
0.16% kworker/2:2-eve [kernel.vmlinux] [k] skb_release_data
0.16% ksoftirqd/2 [kernel.vmlinux] [k] _initial_num_blocks_is_117529
0.16% ksoftirqd/2 [kernel.vmlinux] [k] crypto_aead_encrypt
0.16% ksoftirqd/2 [virtio_net] [k] page_to_skb
0.15% ksoftirqd/2 [kernel.vmlinux] [k] __dev_queue_xmit
0.15% ksoftirqd/2 [tempesta_fw] [k] tfw_hpack_decode
0.15% ksoftirqd/2 [virtio_net] [k] start_xmit
0.15% ksoftirqd/2 [kernel.vmlinux] [k] _encrypt_by_8_new2497
0.15% ksoftirqd/2 [tempesta_fw] [k] tfw_http_conn_msg_alloc
0.15% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_enable_cb_delayed
0.15% ksoftirqd/2 [kernel.vmlinux] [k] __x86_indirect_thunk_rax
0.15% ksoftirqd/2 [kernel.vmlinux] [k] ip_send_check
0.14% kworker/2:2-eve [kernel.vmlinux] [k] __list_del_entry_valid
0.13% ksoftirqd/2 [nf_conntrack] [k] nf_ct_seq_offset
0.13% ksoftirqd/2 [kernel.vmlinux] [k] memset_erms
0.13% ksoftirqd/2 [kernel.vmlinux] [k] __inc_numa_state
0.13% ksoftirqd/2 [kernel.vmlinux] [k] __netif_receive_skb_core.constprop.0
0.13% ksoftirqd/2 [tempesta_fw] [k] tfw_vhost_lookup_default
0.13% ksoftirqd/2 [nf_conntrack] [k] nf_conntrack_in
0.13% ksoftirqd/2 [kernel.vmlinux] [k] kfree
0.12% ksoftirqd/2 [kernel.vmlinux] [k] virtqueue_get_buf
0.12% ksoftirqd/2 [kernel.vmlinux] [k] _zero_cipher_left17529
0.12% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock_bh
0.12% ksoftirqd/2 [kernel.vmlinux] [k] sg_next
0.12% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_find_stream
0.12% ksoftirqd/2 [tempesta_tls] [k] ttls_aead_req_alloc
0.12% ksoftirqd/2 [kernel.vmlinux] [k] rb_next
0.12% ksoftirqd/2 [kernel.vmlinux] [k] ___slab_alloc
0.11% ksoftirqd/2 [kernel.vmlinux] [k] nf_hook_slow
0.11% ksoftirqd/2 [kernel.vmlinux] [k] tcp_v4_rcv
0.11% ksoftirqd/2 [tempesta_fw] [k] tfw_http_tbl_action
0.10% ksoftirqd/2 [kernel.vmlinux] [k] crypto_stats_aead_decrypt
0.10% ksoftirqd/2 [nf_nat] [k] nf_nat_inet_fn
0.10% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.10% ksoftirqd/2 [tempesta_fw] [k] tfw_wq_pop_ticket
0.10% ksoftirqd/2 [kernel.vmlinux] [k] ktime_get_real_ts64
0.10% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_frame_recv
0.09% ksoftirqd/2 [kernel.vmlinux] [k] __mod_timer
0.09% kworker/2:2-eve [kernel.vmlinux] [k] _get_AAD_rest815034
0.09% ksoftirqd/2 [kernel.vmlinux] [k] kmem_cache_alloc
0.09% kworker/2:2-eve [kernel.vmlinux] [k] get_page_from_freelist
0.09% ksoftirqd/2 [kernel.vmlinux] [k] skb_release_head_state
0.09% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest82
0.09% ksoftirqd/2 [kernel.vmlinux] [k] tcp_rcv_established
0.08% ksoftirqd/2 [kernel.vmlinux] [k] free_pcppages_bulk
0.08% ksoftirqd/2 [kernel.vmlinux] [k] consume_skb
0.08% ksoftirqd/2 [kernel.vmlinux] [k] _get_AAD_rest_final15034
0.08% swapper [kernel.vmlinux] [k] pvclock_clocksource_read
0.08% ksoftirqd/2 [tempesta_fw] [k] tfw_gfsm_move
0.08% ksoftirqd/2 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.08% ksoftirqd/2 [tempesta_fw] [k] frang_http_req_handler
0.08% ksoftirqd/2 [kernel.vmlinux] [k] _partial_done29982
0.08% ksoftirqd/2 [tempesta_tls] [k] ttls_encrypt
0.07% ksoftirqd/2 [kernel.vmlinux] [k] __page_cache_release
0.07% ksoftirqd/2 [tempesta_fw] [k] tfw_h2_resp_fwd
0.07% kworker/2:2-eve [kernel.vmlinux] [k] aesni_gcm_init_avx_gen4
0.07% swapper [kernel.vmlinux] [k] key_128_enc_update4
0.07% ksoftirqd/2 [tempesta_fw] [k] tfw_capolicy_match
0.07% ksoftirqd/2 [kernel.vmlinux] [k] ip_rcv_finish_core.constprop.0
0.07% ksoftirqd/2 [tempesta_fw] [k] tfw_http_conn_msg_free
0.07% ksoftirqd/2 [kernel.vmlinux] [k] policy_node
0.07% ksoftirqd/2 [kernel.vmlinux] [k] skb_put
0.07% ksoftirqd/2 [kernel.vmlinux] [k] vring_map_one_sg
0.07% ksoftirqd/2 [kernel.vmlinux] [k] tcp_event_new_data_sent
0.07% ksoftirqd/2 [kernel.vmlinux] [k] __slab_free
0.07% ksoftirqd/2 [tempesta_fw] [k] ss_send
1.24% ksoftirqd/2 [kernel.vmlinux] [k] rb_erase here is most likely related to out-of-order internal TCP queue processing, not HTTP/2 streams processing.
Samples: 262K of event 'cycles', Event count (approx.): 199480018345
Children Self Command Shared Object Symbol
- 1.11% 1.11% ksoftirqd/2 [kernel.vmlinux] [k] rb_erase ◆
1.11% ret_from_fork ▒
kthread ▒
smpboot_thread_fn ▒
run_ksoftirqd ▒
- __do_softirq ▒
- 1.11% net_rx_action ▒
- 1.10% virtnet_poll ▒
- napi_gro_receive ▒
- 1.10% gro_normal_one ▒
netif_receive_skb_list_internal ▒
__netif_receive_skb_list_core ▒
__netif_receive_skb_core.constprop.0 ▒
ip_local_deliver_finish ▒
ip_protocol_deliver_rcu ▒
tcp_v4_rcv ▒
- tcp_v4_do_rcv ▒
- 1.10% tcp_rcv_established ▒
- 1.09% tcp_ack ▒
rb_erase ▒
0.04% 0.04% kworker/2:2-eve [kernel.vmlinux] [k] rb_erase ▒
0.00% 0.00% sh [kernel.vmlinux] [k] __rb_erase_color