Random spikes in dynamodb update latency

[nuel77]

trafficstars

Describe the bug

We are experiencing random latency spikes (which go up to 10s) in DynamoDB operations while using the AWS Rust SDK in our project. These spikes randomly and seem to increase in frequency with increasing time. A weird thing we observed is that the latency doest seem to correleate with the load for example updating 100 items takes 50ms while updating 10 items might sometimes take 100+ms . Despite our efforts to debug the issue, we have been unable to identify the cause. Athough we are concerned if cloning the aws_sdk_dynamodb::client for each upload task is causing any deadlock situation in the library code. would be great to get some pointers here for us to work on.

Expected Behavior

Latency to be fairly consitent and but increasing depending on load .

Current Behavior

Random spikes in latency even for smaller loads.

Reproduction Steps

latency graph from cloud watch:

code snippet for uploader task: https://gist.github.com/nuel77/a48fee5172abdf47efeba14bdaaff3b7

also, our client looks like this:

 pub async fn new(tables: TableConfig, config: &SdkConfig) -> Self {
        let dynamo_db_client = Client::new(config);
        Self {
            client: dynamo_db_client,
            balances_table: tables.balances_table,
            orders_table: tables.orders_table,
            trades_table: tables.trades_table,
            accounts_table: tables.accounts_table,
            transactions_table: tables.transactions_table,
            general_table: tables.general_table,
        }
    }

Possible Solution

No response

Additional Information/Context

No response

Version

(base) ~/CLionProjects/orderbook git:[add-withdrawal-tests]
cargo tree | grep aws-
├── aws-config v1.4.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.6.1
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.10
│   │   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.10 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   │   │   ├── aws-smithy-types v1.1.10 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-types v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   │   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-sdk-sso v1.25.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-runtime v1.5.3
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-ssooidc v1.26.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.3 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-sts v1.25.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-runtime v1.5.3 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.3 (*)
│   ├── aws-smithy-runtime-api v1.6.1 (*)
│   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-dynamodb v1.28.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.3 (*)
│   ├── aws-smithy-runtime-api v1.6.1 (*)
│   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-types v1.2.1 (*)
│   ├── aws-config v1.4.0 (*)
│   ├── aws-sdk-dynamodb v1.28.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-dynamodb v1.28.0 (*)
│   ├── aws-creds v0.36.0
│   ├── aws-region v0.25.4
├── aws-config v1.4.0 (*)
│   ├── aws-config v1.4.0 (*)
│   ├── aws-sdk-s3 v1.29.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-sigv4 v1.2.1 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-checksums v0.60.8
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.3 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-smithy-xml v0.60.8 (*)
│   │   ├── aws-types v1.2.1 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-kms v1.26.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.3 (*)
│   ├── aws-smithy-runtime-api v1.6.1 (*)
│   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-dynamodb v1.28.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-kms v1.26.0 (*)
├── aws-sdk-s3 v1.29.0 (*)
│   ├── aws-config v1.4.0 (*)
│   ├── aws-sdk-timestreamquery v1.26.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.3 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-timestreamwrite v1.25.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.3 (*)
│   │   ├── aws-smithy-runtime-api v1.6.1 (*)
│   │   ├── aws-smithy-types v1.1.10 (*)
│   │   ├── aws-types v1.2.1 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-config v1.26.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.3 (*)
│   ├── aws-smithy-runtime-api v1.6.1 (*)
│   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-kms v1.26.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-config v1.26.0 (*)
├── aws-sdk-dynamodb v1.28.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-config v1.26.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-cloudwatch v1.29.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-query v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.3 (*)
│   ├── aws-smithy-runtime-api v1.6.1 (*)
│   ├── aws-smithy-types v1.1.10 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-dynamodb v1.28.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-config v1.26.0 (*)
├── aws-config v1.4.0 (*)
├── aws-sdk-dynamodb v1.28.0 (*)

Environment details (OS name and version, etc.)

ubunutu

Logs

No response