[camel-aws-ddb-streams-source-kafka-connector] Gets stuck with error java.util.ConcurrentModificationException in logs

[aonamrata]

Hi,

I am using camel-aws-ddb-streams-source-kafka-source-connector V4.8.0 with sts-2.30.2.jar for credentials. We were having issues where the connector will show status "Running" but would be stuck and did not process any new streams messages. we have to explicitly restart the connector to start processing again. We turned on DEBUG logs for the connector and this is what we found

The connector does not die, it's still running. Also there are some new logs after this WARN Failed polling endpoint: DdbStreamEndpoint java.util.ConcurrentModificationException that shows 200 status in logs but not really getting new messages.

[2025-01-22 03:03:30,568] DEBUG Received successful response: 200, Request ID: 6VJEESJK6TF9S2LKBDOF9UNVDBVV4KQNSO5AEMVJF66Q9ASUAAJG, Extended Request ID: not available (software.amazon.awssdk.requestId)

This error normally occurs ~40-50 minutes after the connector is restarted. Nothing gets processed after this time even though the logs don't have any errors or exceptions. I am not 100% sure if it is this error that is causing it but this is the only error/warn in the logs.

This is my connector configuration:

{
  "connector.class": "org.apache.camel.kafkaconnector.awsddbstreamssource.CamelAwsddbstreamssourceSourceConnector",
  "transforms.CamelTypeConverterTransformer.type": "org.apache.camel.kafkaconnector.transforms.CamelTypeConverterTransform$Value",
  "transforms.CamelTypeConverterTransformer.target.type": "java.lang.String",
  "camel.kamelet.aws-ddb-streams-source.table": "qa_pp_identity",
  "camel.kamelet.aws-ddb-streams-source.region": "us-east-1",
  "kafka.bootstrap.servers": "b-1......:9094",
  "topics": "cdc.ppIdentity",
  "transforms": "CamelTypeConverterTransformer",
  "name": "pp_identity_ddbs_src",
  "camel.kamelet.aws-ddb-streams-source.streamIteratorType": "FROM_START",
  "camel.kamelet.aws-ddb-streams-source.useDefaultCredentialsProvider": "true"
}

[oscerd]

I'll try to reproduce in plain camel with the Kamelet and let you know.