dynamodump icon indicating copy to clipboard operation
dynamodump copied to clipboard

Published 20 hours ago • verified publisher icon mifi

How do I improve the speed of the dynamodump import to the target Dynamo DB?

Open cvsudheer108 opened this issue 3 years ago • 6 comments
trafficstars

Hello, I have exported data from the source DynamoDB on to a AWS EC2 machine (t3.small) and then trying to import the data to the target Dynamo DB. I see that it is taking time (1 min for 1 MB ), which might be too much when I add on all my table data. Is there a way to improve the speed of import? I see that there I an option to mention the throughput in : dynamodump wipe-data --throughput, but not for was import-data. Please let me know.

cvsudheer108 avatar Jun 27 '22 17:06 cvsudheer108

I need to migrate the data from one prod account to another and hence can't afford much down time... please advise..

cvsudheer108 avatar Jun 28 '22 03:06 cvsudheer108

1.dynamodata 5MB 1.5 min
2.dynamodata 12MB 7 min 3.dynamodata 32MB 25 min I see the speed reduces as the size of the file increases..

cvsudheer108 avatar Jun 28 '22 05:06 cvsudheer108

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/backuprestore_HowItWorks.html

On Tue, 28 Jun 2022 at 12:31 Sudheer Kumar @.***> wrote:

1.dynamodata 5MB 1.5 min 2.dynamodata 12MB 7 min 3.dynamodata 32MB 25 min I see the speed reduces as the size of the file increases..

— Reply to this email directly, view it on GitHub https://github.com/mifi/dynamodump/issues/28#issuecomment-1168242748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOHVJGLWGLOPG36OPCOHLVRKE2BANCNFSM5Z7KR7OQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Lusitaniae avatar Jun 28 '22 05:06 Lusitaniae

I have resolved the issue usign the AWS Backup service. They have a new feature that enables cross account and cross region back-up of Dynamo DB tables. The current Dynamo DB replication supports only the cross-region feature, but not corss account. This video talks about AWS backup settings in detail : https://www.youtube.com/watch?v=dCy7ixko3tE For DDB backup, we need to enable the following in AWS Backup -> Settings:

  1. Advanced features for Amazon DynamoDB backups - Enable
  2. Cross account-management->Cross-account backup - Enable

cvsudheer108 avatar Jun 29 '22 09:06 cvsudheer108

hi. there should be the --throughput flag for import also. Does it not work?

https://github.com/mifi/dynamodump/blob/093058097b3ba6e008c5ac48bd90ab9fbc1dbaaf/cli.js#L290

mifi avatar Jun 29 '22 19:06 mifi

I think it needs to rewritten to use aws-sdk-v3 instead, like I just did with cognito-backup: https://github.com/mifi/cognito-backup/commit/431e70723a95ecf29d67634f52f5ec2eacfef41e

mifi avatar Jul 02 '22 06:07 mifi