s3cmd
s3cmd copied to clipboard
s3tools
would be nice to support multiple accounts in config
A common pattern I see with s3cmd is to maintain multiple config files like this:
http://mikesisk.com/post/s3cmd-with-multiple-accounts
What would be nice is a way to specify multiple sets of keys in s3cmd.config each with an alias, such that s3cmd could be invoked using the alias as an argument to select that account.
Or perhaps that is overkill. What do people think?
I would also like to store credentials from multiple AWS accounts in a single .s3cfg file.
I agree there should be a way to support multiple accounts.
Perhaps maybe checking for a local config before using the global one in the home folder? This would allow various projects to use s3cmd as normal with different accounts and not complicating or changing the config format.
Also, this can be used to support sync between two S3 buckets more transparently between two different accounts. All example usages I find about sync assume that both S3 buckets have the same access-key-ids and secrets:
https://mdahlman.wordpress.com/2013/12/05/copy-files-between-s3-buckets/
I guess that this is harder to implement in a transparent way than it seems, because:
http://www.patcup.com/copy-contents-in-s3-bucket-between-aws-accounts-with-s3cmd/
One needs to define bucket policies in both endpoints beforehand... I guess that this step could be automated as well.
The sync (remote to remote copy) only sends the command to the destination server using your access_key (valid on the destination server). That key must also have read access to the objects in the source bucket. The destination server then copies the data from the source bucket directly. There's only one set of keys in use during this operation.
On Wed, Feb 4, 2015 at 3:47 AM, Roman Valls Guimerà < [email protected]> wrote:
Also, this can be used to support sync between two S3 buckets more transparently between two different accounts. All example usages I find about sync assume that both S3 buckets have the same access-key-ids and secrets:
https://mdahlman.wordpress.com/2013/12/05/copy-files-between-s3-buckets/
— Reply to this email directly or view it on GitHub https://github.com/s3tools/s3cmd/issues/104#issuecomment-72826657.
Mike has a nice solution to using multiple accounts with s3cmd. http://mikesisk.tumblr.com/post/8703449578/s3cmd-and-multiple-accounts
Why not have s3cmd support the new standard .aws/credentials file? http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs
I'd consider a patch to accept the .aws/credentials file as valid input.
On Thu, Apr 16, 2015 at 10:43 AM, Eyad Sibai [email protected] wrote:
+1 @adamberenzweig https://github.com/adamberenzweig
— Reply to this email directly or view it on GitHub https://github.com/s3tools/s3cmd/issues/104#issuecomment-93767392.
+1, multiple configs would be my preference perhaps s3cmd configure myConfig1 or the like.
Something that could be done is that you create a .s3cfg file without access_key and secret_key. And set them per your needs through env variables: AWS_ACCESS_KEY and AWS_SECRET_KEY you can probably have an alias or little bash script that does that for you.
Otherwise, I'm not sure that a "profile" option would have much sense. What is more convenient to type? Something like: s3cmd -c s3cfg_florent ls s3cmd -c s3cfg_corporate ls or s3cmd -c s3cfg -p florent s3cmd -c s3cfg -p corporate
The later will create more configuration confusion for basic users I think?
The point being that from a first view, you can think that in profile should go: "access_key", "secret_key", "access_token"
But some other people that use encryption or servers that are not aws s3 could consider that the following are also like "profiles":
- kms_key
- encrypt -gpg_passphrase
- use_https
- signature_v2
- host_base
- host_bucket
So, in the end you will want to override most of the fields that are commonly modified in s3cfg file. But that is just my opinion ;-)
What is the purpose of the [default] statement at the top of .s3cfg files, if you can't have more than 1 section and choose between them by specifying the profile?
A workaround is to use update-alternatives.
Use different file names for your .s3cfg file, backup and remove your .s3cfg file and it will be replaced with a symlink to the alternative. So with two accounts and files named .s3cfg-aws-1, .s3cfg-aws-2.
Install syntax update-alteratives --install <link> <name> <path> <priority>
Install the alternatives with:
sudo update-alternatives --install /home/username/.s3cfg s3cfg /home/username-aws-1 20
sudo update-alternatives --install /home/username/.s3cfg s3cfg /home/username-aws-2 10
The alternative with the highest priority will be the default.
Choose alternative with:
sudo update-alternatives --config s3cfg
List or display alternatives with:
sudo update-alternatives --list s3cmd Works on Debian 10. --list may not be in all distros.
Or to get the priority:
sudo update-alternatives --display s3cmd
Test with something like s3cmd ls to see which account you are connected to.
There's a PR that partly addresses this issue; https://github.com/s3tools/s3cmd/pull/995
@noelspringer Pardon me if I'm wrong, but you don't really need all for that to use 2 different config files. By default, s3cmd look for the file: ~/.config/.s3cfg, but you can easily use another configuration file with the "-c" command line flag. Ex: s3cmd -c myfolder/aws-1.s3cfg ls s3cmd -c myfolder/aws-2.s3cfg ls
@sveneh I'm trying to understand what is the exact use case that would need to be improved. Could you tell me what is the exact issue/feature that you would like? To my understanding, this PR will not change a behavior related to multiple accounts.
What I can see that could be needed:
- AWSPROFILE: to use one of the profile of an awscredential file that we import
- Multiple profile for the s3cfg config file itself
For that later case, I'm not really sure of what is the usage/need. Is having 2 configs in 1 single file, so much more interesting than 2 configs inside 2 different config files? Maybe most of the people would like to have the multiple keys inside a single file? maybe a different one than the config file itself? Also, maybe what would be needed is in fact to be able to select a config file with an ENV var? and so that would probably give the impression to use different profiles?
I can't speak for OP, but for me it's related to sync. Syncing between 2 separate accounts or even s3 providers (with different credentials) is a use-case that's currently not covered at all (as far as my google-fu can tell). Tho @mdomsch his reply earlier in the thread suggests that this wouldn't solve that and maybe that deserves its own issue.
Agreed, I was really surprised there isn't a way to configure multiple accounts with their own keys and host_base for use when sync'ing between two different buckets.
FWIW rclone lets you configure independent endpoints and keys for e.g. an s3east and s3west then rclone copy s3east:bucket s3west:otherbucket uses only those credentials when reading from source and writing to destination. Unfortunately rclone does not copy metadata which is why I was hoping to use s3cmd.
@trahloff you can use -c <configFile> to switch between single other accounts, but not use multiple for a command (such as copy/sync), I use rclone for those other commands.
https://linuxamination.blogspot.com/2017/12/s3cmd-configure-multiple-s3-accounts-on.html
Very interesting discussion. I also assumed, that a "section" could be used as profile in order to have several configs in one file as alternative option to have multiple files with one config in each. My main consideration was to use the same ".cfg" file for s3cmd and an server application, which will in my opinion benefit from having only 1 file to read all S3 profiles. For s3cmd on its own, I have to agree, that there is not that much difference in:
s3cmd -c file1
s3cmd -c file2
s3cmd -c file3
or
s3cmd -profile p1
s3cmd -profile p2
s3cmd -profile p3
So, I will go to use a different format ...
