AWS SigV4 for non-service endpoints

[endgame]

At $WORK, we need to create AWS Signature Version 4 requests for non-AWS endpoints. This is because we're connecting to a service which uses an AWS API Gateway and does access control via IAM.

Because Amazonka's signature code is closely tied to its Service record (which identifies an AWS service endpoint, signing options, etc.), we've had to start building our own library. This seems like a problem that's solvable in a way that amazonka (and other AWS libraries) could use, which would cut down on repeated code.

@brendanhay how do you feel about amazonka-core depending on some other aws-request-signers library? It probably means moving all the various AccessKey/SecretKey etc newtypes across (and re-exporting them from their old home for compatibility), as well as moving/renaming Amazonka.Data.Sensitive (which might want to go into a separate data-sensitive library, if we were so inclined), but it's probably good for the ecosystem overall. amazonka-core would then still use its Service type to pass parameters to the signature functions.

If you're nervous about depending on having a critical package like that under someone else's control I could probably set it up as an amazonka contribution, keeping the aws-request-signers name or calling it something like amazonka-core-signers (though I like the former, as it's clearer that it's for non-amazonka libs too.

[endgame]

It's looking more likely that we'll shoe-horn the service we're talking to into an amazonka Service than successfully extract the signing code - the signer is pretty battle-hardened at this point and it seems less likely to introduce breakage.

[brendanhay]

I'm against the idea of depending on anything external to the repository for anything AWS specific, particularly signing code.

Fine with splitting out a separate library if necessary under lib/, less so with custom (non-amazonka prefixed) names only to solve discovery/advertising. (Open to being convinced.)

[m4dc4p]

At my workplace, we started using amazon's IAM support for RDS authentication, which essentially meant the "password" passed to postgres was a signed request. There is a library out there, but I found it didn't work at all (maybe it did at some point the past). I similarly had to abandon the Service record and build out my own request, based on studying the internal of those modules. This is a sample of what the code looks like:

-- | Create a token for authenticating with an RDS instance. The token will be valid for 15
-- minutes after the given signing time.
createToken ::
  -- | AWS Authentication Info
  AuthEnv ->
  -- | AWS Region
  Region ->
  -- | Time to sign token at ()
  UTCTime ->
  -- | Database host
  Text ->
  -- | Database port
  Int ->
  -- | Database username
  Text ->
  -- | Resulting token.
  Text
createToken authEnv region signingTime dbHost dbPort userName =
  let serviceSigningName :: Lens' Service ByteString
      serviceSigningName = lens _serviceSigningName (\s a -> s {_serviceSigningName = a})
      emptyBodySigner :: AWS.Seconds -> AWS.Request a -> AuthEnv -> Region -> UTCTime -> Signed a
      emptyBodySigner expires rq a r ts =
        let meta = signMetadata a r ts presigner digest (prepare rq)
            auth = clientRequestQuery <>~ ("&X-Amz-Signature=" <> toBS (metaSignature meta))
            presigner c shs =
              pair (CI.original hAMZAlgorithm) algorithm
                . pair (CI.original hAMZCredential) (toBS c)
                . pair (CI.original hAMZDate) (Time ts :: AWSTime)
                . pair (CI.original hAMZExpires) expires
                . pair (CI.original hAMZSignedHeaders) (toBS shs)
                . pair (CI.original hAMZToken) (toBS <$> _authSessionToken a)
            -- This is the hash of an empty string, an must be used when creating these RDS
            -- tokens. Required by the V4 signing spec. SEe
            -- https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html and
            -- in particular, note step 6: "If the payload is empty, use an empty string as the input to the hash function."
            digest = Tag "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
            prepare = AWS.requestHeaders %~ (hdr hHost (encodeUtf8 dbHost <> ":" <> toBS dbPort))
         in signRequest meta mempty auth
      req =
        Request
          { _requestService =
              RDS.defaultService & (serviceEndpoint . endpointHost) .~ (encodeUtf8 dbHost)
                & (serviceEndpoint . endpointPort) .~ dbPort
                & serviceSigningName .~ "rds-db"
                & serviceSigner .~ ((RDS.defaultService ^. serviceSigner) {signerPresign = emptyBodySigner}),
            _requestMethod = Method.GET,
            _requestPath = rawPath @Text "/",
            _requestQuery = pair @Text "Action" "connect" ("DBUser" =: userName),
            _requestHeaders = [],
            _requestBody = ""
          }
      toToken (Signed {..}) = decodeUtf8 $ host signedRequest <> ":" <> encodeUtf8 (pack (show (port signedRequest))) <> path signedRequest <> queryString signedRequest
   in toToken $ requestPresign tokenTTL req authEnv region signingTime

[endgame]

There is a library out there, but I found it didn't work at all (maybe it did at some point the past).

I don't think it ever did, as until https://github.com/brendanhay/amazonka/pull/767 amazonka seems to have treated empty bodies as if they were representing an unsigned payload going to S3. I hope to merge it soon.

Thanks for the detailed writeup. I did manage to create a Service record in my use-case. I will leave this issue open until I get a change to write up request signing somewhere accessible.

[ncaq]

I ran API Gateway (HTTP API) on a custom domain and put IAM authentication on it, but I had a lot of trouble connecting with amazonka. Since it was a custom domain, I was not sure of the signature key, so I had to investigate amazonka to know the internal process.

At any rate, the following code seems to be working, and I would like to get this code incorporated if possible, but I am not sure which module I should provide it to.

{-# LANGUAGE FlexibleContexts  #-}
{-# LANGUAGE NamedFieldPuns    #-}
{-# LANGUAGE NoImplicitPrelude #-}
{-# LANGUAGE OverloadedStrings #-}
module AWS.ApiGatewayExecute.Sign (awsSignToHeader) where

import           Amazonka                               as AWS
import           Amazonka.APIGatewayManagementAPI.Types as AWS.ApiGwMaApi
import qualified Network.HTTP.Client                    as N
import qualified Network.HTTP.Types.Method              as N
import           RIO
import qualified RIO.ByteString                         as B
import           Servant.Client

-- | 純粋空間で引数から認証済みの`Request`を生成する。
awsSignToHeader
  :: ToBody body
  => AuthEnv -> UTCTime -> BaseUrl -> Region -> body -> N.Request -> N.Request
awsSignToHeader authEnv signingTime baseUrl region body req =
  let awsRequest = httpRequestToAwsRequest baseUrl region body req
      Signed _meta signedReq = requestSign awsRequest authEnv region signingTime
  in signedReq

-- | `Network.HTTP.Client.Request`のデータを変換して`Amazonka.Request`にして`requestSign`で署名する準備を整える。
-- `Network.HTTP.Client.Request`が`Network.HTTP.RequestBody`を含んでいるのに、
-- `body`を引数で別に取っているのは重複しているように見えるが、
-- `Network.HTTP.RequestBody`はバリアントによって純粋空間で取得できるとは限らないため、
-- 呼び出し側でどうにか出来る余地を残す。
httpRequestToAwsRequest
  :: ToBody body
  => BaseUrl -> Region -> body -> N.Request -> AWS.Request a
httpRequestToAwsRequest baseUrl region body req =
  AWS.Request
  { _requestService = baseUrlExecuteApiService baseUrl region
  , _requestMethod = fromRight (error "Could not parse request method.") $ N.parseMethod $ N.method req
  , _requestPath = rawPath $ N.path req
  , _requestQuery = parseQueryString $ fromMaybe "" $ B.stripPrefix "?" $ N.queryString req
  , _requestHeaders = N.requestHeaders req
  , _requestBody = toBody body
  }

-- | `BaseUrl`からある程度導出出来るカスタムドメインでのサービス定義。
baseUrlExecuteApiService :: BaseUrl -> Region -> Service
baseUrlExecuteApiService BaseUrl{baseUrlScheme, baseUrlHost, baseUrlPort} =
  customDomainExecuteApiService (fromString baseUrlHost) (baseUrlScheme == Https) baseUrlPort

-- | カスタムドメインでのAPI Gatewayのサービス定義。
customDomainExecuteApiService :: ByteString -> Bool -> Int -> Region -> Service
customDomainExecuteApiService customEndpointHost customEndpointSecure customEndpointPort region =
  executeApiService $ const $ Endpoint
  { _endpointHost = customEndpointHost
  , _endpointSecure = customEndpointSecure
  , _endpointPort = customEndpointPort
  , _endpointScope = toBS region
  }

-- | API GatewayのIAM認証突破を行うためのサービス定義。
-- API Gatewayではカスタムドメインが定義できるため、`Endpoint`は引数で定義出来るようにする。
executeApiService :: (Region -> Endpoint) -> Service
executeApiService endpoint =
  Service
  { _serviceAbbrev = "APIGatewayExecuteAPI"
  , _serviceSigner = _serviceSigner AWS.ApiGwMaApi.defaultService
  , _serviceEndpointPrefix = "execute-api"
  , _serviceSigningName = "execute-api"
  , _serviceVersion = _serviceVersion AWS.ApiGwMaApi.defaultService
  , _serviceEndpoint = endpoint
  , _serviceTimeout = Just 70
  , _serviceCheck = statusSuccess
  , _serviceError = parseJSONError "APIGatewayExecuteAPI"
  , _serviceRetry = _serviceRetry AWS.ApiGwMaApi.defaultService
  }

[rickeyski]

I wanted to bump this issue to see if there is anything I can do to help. We are running into this issue because I am trying to remove our usage of aws-iam-authenticator for EKS. The aws-iam-authenticator repo show a python snippet of what is needed to provide the signed url to authenticate to the EKS cluster. However, when I attempt to recreate it with Amazonka, my output is invalid because Action and Version are not params. See below for the difference

Amazonka generated via presignWithHeaders based off of GetCallerIdentity object

https://sts.amazonaws.com/?X-Amz-Algorithm=AWS4-HMAC-SHA256&...

vs the token which is generated via aws-iam-authenticator

https://sts.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&...

Thanks, Rickey