Server-Side Bucket to Bucket

[geoffjukes]

trafficstars

Hi,

Curious to know if copying between buckets is a server-side affair, or a download-upload process.

There are 3 bucket-to-bucket scenarios:

1. Same account, same region
2. Same account, across regions
3. Different accounts for above

I’ve done some crude tests, and bucket-to-bucket seems faster than doing separate download-upload processes. So I thought I’d just ask :)

Many thanks,

Geoff

[willmcgugan]

S3FS isn't aware of bucket-to-bucket transfers. So it will essentially do a download/upload.

It should be possible to detect where a bucket-to-bucket transfer would be possible. To be honest, I didn't even know that was possible. Will look in to that...

[geoffjukes]

@willmcgugan I didn't either, until recently, It's actually pretty simple:

    s3.meta.client.copy_object(
        CopySource={
            'Bucket': src,
            'Key': key
        },
        Bucket=tgt,
        Key=new_key if new_key else key,
        StorageClass='STANDARD_IA'
    )

where src and tgt are Boto3 Bucket instances (with thier appropriate connections).

http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.copy

Thanks for looking into it!

[ELind77]

Any progress on this?

Any guidance on the change that would need to be made? Currently, it looks like copy is doing the right thing, but it's not really clear to me what happens when the src_path is an S3FS object from another bucket. Is there a way to get the bucket name from the src_path without causing problems or inconsistencies with the API?

Context: https://github.com/PyFilesystem/s3fs/blob/7f95af69e904e58b068ac4f6b39c2da488d5d918/fs_s3fs/_s3fs.py#L790