s3-parallel-put
s3-parallel-put copied to clipboard
301 Moved Permanently
Executing the command with an S3 bucket located in Sydney Australia (ap-southeast-1) throws an error.
root@vmd001 [/path/to/folder/test]# /path/to/folder/s3-parallel-put --bucket=vmd001 --put=add --insecure --dry-run --limit=1 .
Traceback (most recent call last):
File "/path/to/folder/s3-parallel-put", line 420, in
Got same error.
This is not addressed on the README
, but if you look at the code there's an option to set the AWS region as --bucket_region=YOUR_REGION
.
In my case setting it to my region, eu-west-1
, solved the issue:
./s3-parallel-put --bucket=MY_BUCKET \
--put=stupid
--dry-run --limit=1
--bucket_region=eu-west-1
--insecure
.
Is it fixed yet?
no yet
I found removing the following fixed it for me
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
commenting out calling_format = boto.s3.connection.OrdinaryCallingFormat(),
seems to result in a 400 being returned for me. any word on a work around for this?
After some investigation we narrowed down the issue to https://github.com/mishudark/s3-parallel-put/blob/4121ccea2cd03871f6d68ba1853cf19d27f94f54/s3-parallel-put#L289
The initial connection uses boto.s3.connect_to_region
but the child processes do not.
Swapping this line for
connection = boto.s3.connect_to_region(options.bucket_region,
is_secure=options.secure,
)
seemed to fix our issue and allow us to upload to a region other than us-east-1.
To use a non-default region you must specify both the --bucket_region
and the --host
, as follows:
s3-parallel-put --bucket=$BUCKET . --bucket_region=ca-central-1 --host=s3.ca-central-1.amazonaws.com
The S3 Endpoints can be found here https://docs.aws.amazon.com/general/latest/gr/s3.html