Occasional Timeout Exception to cloudwatch

[BobbyJohansen]

ocasionally we will see a timeout to cloudwatch with a could not publish 20 dataums

2019-09-20 15:57:59,847 WARN  [cw-metrics-publisher] [software.amazon.kinesis.metrics.CloudWatchMetricsPublisher] [] Could not publish 20 datums to CloudWatch
java.util.concurrent.TimeoutException: null
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
	at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.blockingExecute(CloudWatchMetricsPublisher.java:87)
	at software.amazon.kinesis.metrics.CloudWatchMetricsPublisher.publishMetrics(CloudWatchMetricsPublisher.java:74)
	at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.runOnce(CloudWatchPublisherRunnable.java:138)
	at software.amazon.kinesis.metrics.CloudWatchPublisherRunnable.run(CloudWatchPublisherRunnable.java:84)
	at java.base/java.lang.Thread.run(Thread.java:834)

[eoliphan]

Yeah, seeing this since upgrading from 2.2.1 to 2.2.4

[RommelTJ]

I saw this issue as well. Setting a connectionMaxIdleTime of 5 seconds on the SdkAsyncHttpClient seems to have removed the issue for me.

This is how I configured my netty client:

val nettyClient: SdkAsyncHttpClient =
      NettyNioAsyncHttpClient.builder()
        .connectionMaxIdleTime(Duration.ofSeconds(5))
        .maxConcurrency(100)
        .maxPendingConnectionAcquires(10000)
        .build()

[fabian-dev]

I'm also seeing these warnings in 2.2.6.

Are these retry warnings of the cloud watch client? I can't figure out if subsequent CloudWatch requests suceed and these warnings are only related to the retries.

Has anyone a client configuration that worked for him using the async client from: CloudWatchAsyncClient.builder().build()

[shahrakesh]

We are also seeing these randomly.

Would be good to know if these can be ignored or is it a symptom of some issue?

[nbarabash]

v 2.3.4 Could not publish 13 datums to CloudWatch java.util.concurrent.TimeoutException: null

Today it is the first time I have seen it ever. Weeks long it worked well.

[joshng]

🦗