sysbench
sysbench copied to clipboard
akopytov
different hotspot on libssl/libcrypto from sysbench 1.0 to 1.1
I'm running sysbench 1.0.20 / 1.1.0 on an ARM server, but observed different hotspots on the mysqld side:
(mysql version: 8.0.30)
The v1.0.20 case:
The v1.1.0 case:
The first case obviously has a nonnegligible burden on libssl/libcrypt. Here is the expanded result:
+ 99.99% 64.96% mysqld
+ 99.99% 1.32% libpthread-2.28.so
+ 99.98% 5.34% libc-2.28.so
- 26.58% 1.90% libssl.so.1.1.1k
- 20.32% SSL_write
- 0xffff886a598c
- 20.30% 0xffff886946b8
- 17.37% 0xffff886944a0
- 17.35% 0xffff88693750
- 17.35% BIO_write
- 17.14% 0xffff88451f64
- 17.11% 0xffff88452f30
- 17.05% 0xffff88458fe0
- 17.04% __libc_write
- 16.91% el0_sync
- el0_sync_handler
- 16.88% do_el0_svc
- 16.64% __arm64_sys_write
- 16.63% ksys_write
- 16.46% vfs_write
- 16.35% __vfs_write
- 16.31% new_sync_write
- 16.24% sock_write_iter
- 16.18% sock_sendmsg
- 16.06% inet6_sendmsg
- 15.96% tcp_sendmsg
- 15.70% tcp_sendmsg_locked
- 13.82% tcp_push
- 13.79% __tcp_push_pending_frames
- 13.71% tcp_write_xmit
- 13.11% __tcp_transmit_skb
- 12.46% ip_queue_xmit
- 12.43% __ip_queue_xmit
- 12.29% ip_local_out
- 9.44% ip_output
- 9.11% ip_finish_output
- 9.09% __ip_finish_output
- 9.01% ip_finish_output2
- 7.72% __local_bh_enable_ip
- do_softirq.part.1
- 7.65% __softirqentry_text_start
- 7.53% net_rx_action
- 7.43% __napi_poll
- 7.41% process_backlog
- 7.30% __netif_receive_skb
- 7.29% __netif_receive_skb_core
- 7.08% ip_rcv
- 5.76% ip_rcv_finish
- 5.64% ip_local_deliver
- 5.01% ip_local_deliver_finish
- 4.94% ip_protocol_deliver_rcu
- 4.80% tcp_v4_rcv
- 4.09% tcp_v4_do_rcv
- 3.99% tcp_rcv_established
- 1.66% tcp_ack
1.09% tcp_clean_rtx_queue
- 1.56% tcp_data_ready
- sock_def_readable
- 1.47% __wake_up_sync_key
__wake_up_common_lock
0.54% nf_hook_slow
- 1.16% nf_hook_slow
- 0.95% nft_do_chain_inet
0.94% nft_do_chain
- 1.13% dev_queue_xmit
- 1.12% __dev_queue_xmit
- 0.63% dev_hard_start_xmit
0.51% loopback_xmit
- 2.82% __ip_local_out
- 2.59% nf_hook_slow
- 1.46% ipv4_conntrack_local
- 1.37% nf_conntrack_in
0.68% nf_conntrack_tcp_packet
- 0.70% sk_stream_alloc_skb
0.59% __alloc_skb
- 2.46% 0xffff88693e4c
- 1.80% 0xffff88698824
- 1.79% 0xffff884f8de0
- 1.78% 0xffff884ef2a0
- CRYPTO_gcm128_encrypt_ctr32
1.11% CRYPTO_gcm128_encrypt_ctr32
- 4.08% SSL_read
- 0xffff886a56e0
- 4.05% 0xffff8869b600
- 4.03% 0xffff88694a60
- 2.24% 0xffff88696e8c
- 2.22% 0xffff88693454
- BIO_read
- 1.95% 0xffff88451dd4
- 1.93% 0xffff88452f80
- 1.85% 0xffff88458f44
- 1.84% __libc_read
- 1.67% el0_sync
- el0_sync_handler
- 1.66% do_el0_svc
- 1.38% __arm64_sys_read
- 1.37% ksys_read
- 1.22% vfs_read
- 1.09% __vfs_read
- 1.04% new_sync_read
- 0.97% sock_read_iter
- 0.91% sock_recvmsg
- 0.86% inet6_recvmsg
0.82% tcp_recvmsg
0.94% 0xffff886971d0
- 0.83% 0xffff88697478
- 0.83% 0xffff88693454
- BIO_read
- 0.78% 0xffff88451dd4
- 0.78% 0xffff88452f80
- 0.77% 0xffff88458f44
- 0.77% __libc_read
- 0.69% el0_sync
- el0_sync_handler
- 0.69% do_el0_svc
- 0.59% __arm64_sys_read
- ksys_read
- 0.54% vfs_read
- 0.51% __vfs_read
0.50% new_sync_read
- 1.90% thread_start
start_thread
pfs_spawn_thread
handle_connection
- do_command
- 0.95% dispatch_command
- THD::send_statement_status
- 0.85% Protocol_classic::send_eof
net_send_ok
net_flush
net_write_packet
vio_ssl_write
- SSL_write
- 0.75% 0xffff886a598c
0.63% 0xffff886946b8
- 0.95% Protocol_classic::get_command
my_net_read
net_read_packet
net_read_raw_loop
- vio_ssl_read
- 0.88% SSL_read
- 0.75% 0xffff886a56e0
- 0.69% 0xffff8869b600
0.53% 0xffff88694a60
+ 24.24% 3.77% libcrypto.so.1.1.1k
+ 22.19% 22.19% [kernel.kallsyms]
0.34% 0.34% [vdso]
0.28% 0.16% libstdc++.so.6.0.30
0.01% 0.01% libm-2.28.so
There is a CRYPTO_gcm128_encrypt_ctr32() function above, seems sysbench v1.0 is communicating with mysqld through AES_128_GCM, I'm not sure about the correct ssl cipher name. Anyway, this plays a part of mysqld hotspots.
After changing to sysbench v1.1 (current master branch) without changing any other configuration like mysql cnf or any environment variables or any sysbench parameters, just change the sysbench binary, the above hotspots on libssl/libcrypto are all gone.
Does anyone knows the change/background on SSL cipher mechnism from sysbench v1.0 to v1.1 ???
