Support loading credentials from standard config / environment (via AWSCredentialsProvider)

[mjwillson]

A small feature request. To be honest this isn't particularly hard to do directly:

(def s3-client
  ;; This looks for credentials in the standard places.
  (AmazonS3Client. (DefaultAWSCredentialsProviderChain.)))

But it'd be nice if this could be achieved via your s3-client function too so you can combine it the clojurey conventions for specifying the ClientConfiguration.

[mjwillson]

Actually I realised the problem is bigger, since all the other functions take a credentials map rather than accept an AmazonS3Client object itself. It'd be nice if it supported passing in the AmazonS3Client directly, and/or some way to specify in the credentials map for it to use one of the standard AWSCredentialsProvider subclasses

[slotrans]

A simple way to do it is by just changing the client construction from: client (AmazonS3Client. aws-creds client-configuration)] to: client (AmazonS3Client. client-configuration)]

That invocation uses the default credentials provider chain. Unfortunately cred is still an argument to every function, but you can pass an empty map and it works. This is the smallest change I can think of that enables proper authentication behavior.

[weavejester]

I feel that I should point out that the DefaultAWSCredentialsProviderChain could easily be represented as a map. We don't need to stoop to using Java objects to represent key-value data.

[slotrans]

In what way is DefaultAWSCredentialsProviderChain key-value data? It's true that it implements AWSCredentialsProvider and therefore provides a getCredentials() method which returns an AWSCredentials, and that object is basically a glorified map. But I don't see how you get from there to "DefaultAWSCredentialsProviderChain could easily be represented as a map".

At any rate, the data is not actually the interesting part of DefaultAWSCredentialsProviderChain, it's the behavior that's interesting, in terms of checking specific standard places for credentials in a specific standard order. This really matters to AWS users, especially those who have standardized on IAM instance roles for authentication.

My interest in this comes from being a user of github.com/factual/drake, which uses clj-aws-s3 for S3 support. And because clj-aws-s3 does not follow best practices for AWS authentication, neither can drake. Implementing a fix in drake itself is possible, but only by using the AWS Java SDK directly, which rather defeats the purpose of using a library that wraps it.

[weavejester]

What I mean is:

(defn- aws-cred-map [^AWSCredentials aws-creds]
  {:access-key (.getAWSAccessKeyId aws-creds)
   :secret-key (.getAWSSecretKey aws-creds)})

(defn default-aws-creds []
  (-> (DefaultAWSCredentialsProviderChain.) .getCredentials aws-cred-map))

[MichaelBlume]

I don't understand AWS well enough to know why this is, but when I just let it use the credentials object from the provider chain, I can sign S3 requests in such a way that they work, while when I pull an access/secret key out of the credentials and try to just use those, request-signing breaks.

[randomvariable]

Hi @MichaelBlume,

If you're pulling an access/secret key out of a credential derived from an IAM Instance Profile (or any delegated account which is assuming a role from the same or another account via the AWS Security Token Service, you need three pieces to authenticate:

• Access Key
• Secret Key
• Session Token

See http://docs.aws.amazon.com/STS/latest/UsingSTS/CreatingSessionTokens.html for details. As such, it can't be passed into the current functions.

[joelittlejohn]

I think this library is basically deprecated. We should use Amazonica, or something else like https://github.com/cognitect-labs/aws-api, instead. That said, we have some old projects that can't easily be migrated off weavejester/clj-aws-s3.

As such, it can't be passed into the current functions.

This library does actually support session token. Take a look at https://github.com/weavejester/clj-aws-s3/commit/8e36b2dd93526695a582dd412a89f62b1d295ab5.

So I was able to work around the lack of support for the DefaultCredentialsProviderChain in clj-aws-s3 by instantiating the chain myself, calling getCredentials, and building a map with :access-key, :secret-key and :token (as suggested in https://github.com/weavejester/clj-aws-s3/issues/69#issuecomment-84499442).