s3-sync-client icon indicating copy to clipboard operation
s3-sync-client copied to clipboard

Paths Duplicated in S3 -> S3 Sync

Open brianlenz opened this issue 4 months ago • 1 comments

It looks like paths get duplicated unexpectedly when doing an S3 -> S3 sync. In this use case, the sync only includes files in sub-folders of the bucket. For some reason, the path that the files are located in are duplicated unexpectedly in the destination key.

It's easiest to represent through code:

const s3Client = new S3Client({ /* ... */ });
const syncClient = new S3SyncClient({ client: s3Client });
const sourcePath = `s3://bucket-a/x/y/z`;
const destPath = `s3://bucket-b/x/y/z`;
await syncClient.sync(sourcePath, destPath, {
  sizeOnly: true,
  del: true,
  relocations: [
    (key) => {
      // the key here has unexpected duplication of paths in it:
      // x/y/z//x/y/z/filename.txt
      // to fix, relocate it by stripping the duplication from the start.
      // new path: x/y/z/filename.txt
      return key.replace('x/y/z//', '');
    },
  ],
});

In this example, in transferring from x/y/z to x/y/z, the destination folder is actually x/y/z//x/y/z (note the double // in the middle of the path, too).

For now, we've worked around it using relocations as shown above, but it seems like this is probably a bug that should be fixed?

brianlenz avatar Feb 14 '24 16:02 brianlenz