Dozens of WARN DynamoDBJournal at=backoff request=BatchGetItemRequest sleep=8

[sarxos]

Hi @sclasen,

I'm getting dozens of such warnings in the log, but the message they comes with is completely not understandable for me.

Can you please give me a hint on how to get rid of these warnings? Or at least what does it means?

WARN a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2

There are plenty of them:

2015-05-18 16:29:48,376 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-34] [ClusterSystem-akka.actor.default-dispatcher-23] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=unprocessed-reads, unprocessed=59
2015-05-18 16:29:48,425 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-34] [ClusterSystem-akka.actor.default-dispatcher-14] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,428 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-49] [ClusterSystem-akka.actor.default-dispatcher-22] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,429 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-48] [ClusterSystem-akka.actor.default-dispatcher-14] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,437 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-48] [ClusterSystem-akka.actor.default-dispatcher-42] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,440 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-49] [ClusterSystem-akka.actor.default-dispatcher-42] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,445 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-34] [ClusterSystem-akka.actor.default-dispatcher-43] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,445 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-45] [ClusterSystem-akka.actor.default-dispatcher-43] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,446 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-47] [ClusterSystem-akka.actor.default-dispatcher-43] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,447 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-49] [ClusterSystem-akka.actor.default-dispatcher-37] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=2
2015-05-18 16:29:48,506 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-49] [ClusterSystem-akka.actor.default-dispatcher-31] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=4
2015-05-18 16:29:48,509 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-46] [ClusterSystem-akka.actor.default-dispatcher-31] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=4
2015-05-18 16:29:48,517 [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-46] [ClusterSystem-akka.actor.default-dispatcher-28] WARN  a.p.journal.dynamodb.DynamoDBJournal - at=backoff request=BatchGetItemRequest sleep=4

[sarxos]

Ok, I was playing with the Dynamo DB settings in the Amazon Console and it seems like this message is generated when throughput to Dynamo DB table is too low (in my case it was set to 1 r/w capacity). I modified it to 10 r/w and the message is gone. That's good.

However, I will leave this issue open because as I stated above - the message that comes with this warning is completely unintuitive. Imho, this should be rather something like:

Cannot read/write state due to exhausted DynamoDB throughput, wait 4 seconds for next attempt