Timeout on many requests

[sysroute0]

Hello,

When we start generating thousands of KMS requests to XKS proxy, the following logs start appearing in xks-proxy.log and timeout message saying the request can't be executed: 2024-02-23T11:06:54.285096Z DEBUG tokio-runtime-worker hyper::proto::h1::conn: 742: error shutting down IO: Transport endpoint is not connected (os error 107) 2024-02-23T11:06:54.285134Z DEBUG tokio-runtime-worker hyper::proto::h1::conn: 742: error shutting down IO: Transport endpoint is not connected (os error 107) 2024-02-23T11:06:54.285168Z DEBUG tokio-runtime-worker hyper::proto::h1::conn: 742: error shutting down IO: Transport endpoint is not connected (os error 107) 2024-02-23T11:06:54.285202Z DEBUG tokio-runtime-worker hyper::proto::h1::conn: 742: error shutting down IO: Transport endpoint is not connected (os error 107) 2024-02-23T11:06:54.285233Z DEBUG tokio-runtime-worker hyper::proto::h1::conn: 742: error shutting down IO: Transport endpoint is not connected (os error 107) 2024-02-23T11:06:54.285266Z DEBUG tokio-runtime-worker hyper::proto::h1::conn: 742: error shutting down IO: Transport endpoint is not connected (os error 107)

We are running xks-proxy on a EC2 instance which is powerful enough and when we check the monitoring it's not loaded at all. When we load balance the requests behind a NLB and many xks-proxies behind it, this problem doesn't exist. Keep in mind we start many docker containers on the same EC2 instance and load balance between them, which proves the issue is not related with overload of the ec2 itself.

Do you have any ideas what could be the reason for that. Here's the configuration we use in settings.toml: `[server] ip = "0.0.0.0" port = 8000 region = "eu-west-1" service = "kms-xks-proxy"

[server.tcp_keepalive] tcp_keepalive_secs = 60 tcp_keepalive_retries = 3 tcp_keepalive_interval_secs = 1

[tracing] is_stdout_writer_enabled = true is_file_writer_enabled = true level = "DEBUG" directory = "/var/local/xks-proxy/logs" file_prefix = "xks-proxy.log" rotation_kind = "HOURLY"

[security] is_sigv4_auth_enabled = true is_tls_enabled = false is_mtls_enabled = false

[tls] tls_cert_pem = "/var/local/xks-proxy/tls/server_cert.pem" tls_key_pem = "/var/local/xks-proxy/tls/server_key.pem"

[[external_key_stores]] uri_path_prefix = "/keys" sigv4_access_key_id = "XXXXXXXXXXXXXX" sigv4_secret_access_key = "XXXXXXXXXXX" xks_key_id_set = ["abc123", "xyz123"]

[pkcs11] session_pool_max_size = 300 session_pool_timeout_milli = 45 session_eager_close = false user_pin = "" PKCS11_HSM_MODULE = "/usr/local/lib/libvault-pkcs11.so" context_read_timeout_milli = 100

[limits] max_plaintext_in_base64 = 8192 max_aad_in_base64 = 16384

[hsm_capabilities] can_generate_iv = false is_zero_iv_required = false`

Please let me know if you need more information.

Thank you!