dynamodb-replicator icon indicating copy to clipboard operation
dynamodb-replicator copied to clipboard

Order keys for s3

Open btalbot opened this issue 8 years ago • 1 comments

As reported in #90 there is a bug with key ordering when computing S3 keys. The bug reporter submitted a PR #91 which was never accepted. Maybe the use of ES6 features or lack of tests to demonstrate the bug was at issue.

This PR updates the existing tests to use a range key so that the issue is obvious and an ES5 compatible fix to the incremental and backfill functions that are affected.

btalbot avatar May 09 '17 02:05 btalbot

In case it's not clear, this bug causes incremental backups to become corrupted if the order of keys from the event stream ever change or differ from the declaration order used in s3-backfill.

The incremental backup directory will be left with up to two copies of the same DDB item but with different S3 keys. Any snapshots created from the corrupted incremental backup will contain both copies of items; if used to restore a DDB table, both items for the same keys will be loaded with the last one loaded being kept (which is not the same as the desired most recent).

This cannot happen with HASH keyed only items, but any table-stream with a RANGE key is susceptible.

btalbot avatar May 09 '17 18:05 btalbot