openresty icon indicating copy to clipboard operation
openresty copied to clipboard

One random worker consumes 100% cpu

Open dream110fly opened this issue 2 years ago • 2 comments

Hello One random worker of openresty occasionally consumes 100% of the cpu, openresty version is 1.19.3。Use sample-bt tool to catch nginx c backtraces, From the flame graph we can see that ngx_ssl_shutdown take too much cpu time, we also find some errors in error log。 How can we debug this? we use intel QAT with async mode nginx, detail info is from https://github.com/intel/asynch_mode_nginx,nginx config file related to ssl is as follows:

nginx config

load_module modules/ngx_ssl_engine_qat_module.so; lua_ssl_verify_depth 5; ssl_session_timeout 86400; proxy_ssl_name $upstream_host; proxy_ssl_server_name on; ssl_asynch on; ssl_certificate cert/default.pem; ssl_certificate_key cert/default.key; ssl_session_cache shared:SSL:20m; ssl_session_timeout 10m; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:HIGH:!ADH:!EXPORT56:!RC4:!NULL:!aNULL:!DH:!DHE:+MEDIUM; ssl_prefer_server_ciphers on; ssl_session_tickets off;

error log

023/06/28 09:33:20 [crit] 53540#53540: *83943154523 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking, 2023/06/28 09:33:20 [crit] 53517#53517: *83943181752 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking, 2023/06/28 09:33:23 [crit] 53480#53480: *83943257007 SSL_do_handshake() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SS routines:ssl3_get_record:decryption failed or bad record mac) while SSL handshaking, 2023/06/28 09:33:23 [crit] 53487#53487: *83943260352 SSL_read() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac) while waiting for request, 2023/06/28 09:33:46 [crit] 10802#10802: *83944140841 SSL_read() failed (SSL: error:800640A8:lib(128):aes_gcm_tls_cipher:gcm tag verify failure error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac) while keepalive 2023/06/28 09:33:57 [crit] 53509#53509: *83944638838 SSL_do_handshake() failed (SSL: error:141B306E:SSL routines:tls_collect_extensions:bad extension) while SSL handshaking 2023/06/28 09:34:03 [crit] 53570#53570: *83944840894 SSL_do_handshake() failed (SSL: error:141CF0D1:SSL routines:tls_parse_ctos_key_share:missing supported groups extension) while SSL handshaking

flame

image

dream110fly avatar Jun 27 '23 07:06 dream110fly

Need more information about the issue. Please try openresty-xray to analyze this issue.

zhuizhuhaomeng avatar Jun 29 '23 01:06 zhuizhuhaomeng

thank you, I will try openresty-xray

dream110fly avatar Jun 29 '23 01:06 dream110fly