"TrimmedDataAccessException" when working with a vanilla database

[and3rson]

Describe the bug

When fetching records from a shard with a newly created DB with empty tables, the TrimmedDataAccessException exception occurs when first insertion happens in DB.

This is exact same issue as described here: https://stackoverflow.com/questions/38378831/dynamodb-local-streams-getrecords-returns-trimmeddataaccessexception

Expected Behavior

The records from a shard should be returned without any errors.

Current Behavior

botocore.errorfactory.TrimmedDataAccessException: An error occurred (TrimmedDataAccessException) when calling the GetRecords operation: The operation attempted to read past the oldest stream record in a shard.

Reproduction Steps

1. Set up a new local DDB instance
2. Create a new table test
3. Create and run a script that iterates over records from shards:

   stream_arn = '...'
   client = boto3.client('dynamodbstreams', endpoint_url='127.0.0.1:8000')
   shard_id = client.descibe_stream(StreamArn=stream_arn, Limit=100)['StreamDescription']['Shards'][0]['ShardId']
   print('Shard ID:', shard_id)
   iterator = client.get_shard_iterator(StreamArn=stream_arn, ShardId=shard_id, ShardIteratorType='LATEST')['ShardIterator']
   while True:
       response = client.get_records(ShardIterator=iterator, Limit=100)
       sleep(1)
   

4. Insert a record into test
5. Observe exception during client.get_records(...)
6. Restart script
7. Insert another record into "test"
8. No exception is raised, client.get_records(...) now works properly with the same shard ("Shard ID" output same as during previous run)

Same behavior can also be reproduced with this script: https://github.com/MauriceBrg/aws-blog.de-projects/tree/master/dynamodb-streamgazer

1. Set up a new local DDB instance
2. Create a new table test
3. Run python3 dynamodb_streamgazer.py <STREAM_ARN>
4. Insert a record into test
5. Observe exception during client.get_records(...)
6. Restart dynamodb_streamgazer.py
7. Insert another record into "test"
8. No exception is raised, client.get_records(...) now works properly

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

Docker image - amazon/dynamodb-local:1.21.0 (904626f640dc)

JDK version used

Docker image - amazon/dynamodb-local:1.21.0 (904626f640dc)

Operating System and version

ArchLinux host

[debora-ito]

Hey @and3rson apologies for the long delay in replying here.

We are not the owners of the DynamoDB Local library, I'll reroute this to the DynamoDB Local team. Will update here when I get a reply.

[debora-ito]

The DynamoDB Local team made changes to address this issue, it is included in the latest version 2.4.0.

If you have further questions, please reach out to the DynamoDB Local team directly in the repo awslabs/amazon-dynamodb-local-samples

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.