how to use y-sweet server with AWS s3 auth with IAM role?

[jklee84]

Thanks for your huge effort. I tried using y-sweet with S3, and in my environment, only IAM role authentication is accepted.

How can I use the y-sweet server with AWS S3 authentication via IAM role? Can I get any advice?

[paulgb]

Hi @jklee84, the Y-Sweet server itself is not capable of temporary IAM credential refreshing, but you could generate temporary credentials (with AssumeRole) and pass them into the Y-Sweet server. AWS limits these tokens to a 12 hour validity period, so you would have to restart the server with new credentials every 12 hours. Clients will reconnect automatically, so this restart could be done with minimal disruption.

[Dracyr]

We also needed this, are using y-sweet as a long lived service, and didn't want to restart the server every now and then.

Solved it via Proxying over Envoy, which is capable of handling the request signing and using the instance credentials to do so

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 9080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: aws_service
                http_filters:
                  - name: envoy.filters.http.lua
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
                      # Remove the signed query headers from ysweet, as we want to replace them with our own
                      inline_code: |
                        function envoy_on_request(request_handle)
                          local path = request_handle:headers():get(":path")
                          local path_without_query = path:gsub("?.*", "")
                          request_handle:headers():replace(":path", path_without_query)
                        end
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: aws_service
      type: LOGICAL_DNS
      dns_lookup_family: V4_ONLY
      load_assignment:
        cluster_name: aws_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: YOUR_S3_BUCKET_HERE.s3.amazonaws.com
                      port_value: 443
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          upstream_http_protocol_options:
            auto_sni: false
            auto_san_validation: false
          auto_config:
            http2_protocol_options: {}
          http_filters:
          - name: envoy.filters.http.aws_request_signing
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning
              service_name: "s3"
              region: "eu-west-1"
              host_rewrite: YOUR_S3_BUCKET_HERE.s3.amazonaws.com
              use_unsigned_payload: true
              match_excluded_headers:
              - prefix: x-envoy
              - prefix: x-forwarded
              - exact: x-amzn-trace-id
          - name: envoy.filters.http.upstream_codec
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

[emendoza2]

This seems like an issue with how y-sweet internally handles s3 authorization. Currently, the executable pulls from environment variables, but standard projects like the AWS cli can pull from the IMDS on-demand. Would it be possible to use these as a fallback when environment variables are not specified?

[ctriley]

I think the issue is that they are currently using rusty-s3 instead of the official AWS Rust SDK. I'm not sure why this decision was made -- was it to support S3 compatible buckets and not just S3 itself? @paulgb