s3-sync-client
s3-sync-client copied to clipboard
Paths Duplicated in S3 -> S3 Sync
It looks like paths get duplicated unexpectedly when doing an S3 -> S3 sync. In this use case, the sync only includes files in sub-folders of the bucket. For some reason, the path that the files are located in are duplicated unexpectedly in the destination key.
It's easiest to represent through code:
const s3Client = new S3Client({ /* ... */ });
const syncClient = new S3SyncClient({ client: s3Client });
const sourcePath = `s3://bucket-a/x/y/z`;
const destPath = `s3://bucket-b/x/y/z`;
await syncClient.sync(sourcePath, destPath, {
sizeOnly: true,
del: true,
relocations: [
(key) => {
// the key here has unexpected duplication of paths in it:
// x/y/z//x/y/z/filename.txt
// to fix, relocate it by stripping the duplication from the start.
// new path: x/y/z/filename.txt
return key.replace('x/y/z//', '');
},
],
});
In this example, in transferring from x/y/z
to x/y/z
, the destination folder is actually x/y/z//x/y/z
(note the double //
in the middle of the path, too).
For now, we've worked around it using relocations
as shown above, but it seems like this is probably a bug that should be fixed?