teleport icon indicating copy to clipboard operation
teleport copied to clipboard

tsh aws proxy's request size limit is prohibitive

Open programmerq opened this issue 1 year ago • 1 comments

Expected behavior:

When running tsh aws ... commands, they should work with the same sort of actions when not using tsh aws to access aws.

Current behavior:

In particular, updating a lambda function can easily hit the 10 megabyte limit.

$ tsh aws --app myapp lambda  update-function-code --function-name myfunction --zip-file ./file.zip
DEBU             Started forwarding request for "lambda.us-west-2.amazonaws.com:443". alpnproxy/forward_proxy.go:352
INFO [CA]        Generating TLS certificate {0x105287480 0x1400099e120 SERIALNUMBER=222492948881924740888626254140395608531,CN=lambda.us-west-2.amazonaws.com,O=Teleport 2023-03-17 04:04:57 +0000 UTC [lambda.us-west-2.amazonaws.com] [] 5 []}. common_name:lambda.us-west-2.amazonaws.com dns_names:[lambda.us-west-2.amazonaws.com] locality:[] not_after:2023-03-17 04:04:57 +0000 UTC org:[Teleport] org_unit:[] tlsca/ca.go:1037
ERRO [AWS_ACCES] AWS signature verification failed. error:[
ERROR REPORT:
Original Error: *trace.LimitExceededError the read limit is reached
Stack Trace:
        github.com/gravitational/teleport/lib/utils/aws/aws.go:168 github.com/gravitational/teleport/lib/utils/aws.tryDrainBody
        github.com/gravitational/teleport/lib/utils/aws/aws.go:148 github.com/gravitational/teleport/lib/utils/aws.GetAndReplaceReqBody
        github.com/gravitational/teleport/lib/utils/aws/aws.go:195 github.com/gravitational/teleport/lib/utils/aws.VerifyAWSSignature
        github.com/gravitational/teleport/lib/srv/alpnproxy/aws_local_proxy.go:50 github.com/gravitational/teleport/lib/srv/alpnproxy.(*AWSAccessMiddleware).HandleRequest
        github.com/gravitational/teleport/lib/srv/alpnproxy/local_proxy.go:275 github.com/gravitational/teleport/lib/srv/alpnproxy.(*LocalProxy).StartHTTPAccessProxy.func3
        net/http/server.go:2109 net/http.HandlerFunc.ServeHTTP
        net/http/server.go:2947 net/http.serverHandler.ServeHTTP
        net/http/server.go:1991 net/http.(*conn).serve
        runtime/asm_arm64.s:1172 runtime.goexit
User Message: the read limit is reached] alpnproxy/aws_local_proxy.go:51

This appears to run into the limit specified by https://github.com/gravitational/teleport/blob/1a7ccf893c5c89f809b6c1044308d3132ab8a870/constants.go#L505

Bug details:

  • Teleport version: 12.1.1
  • Recreation steps: attempt to run any aws request that will send more than 10 megabytes of data.
  • Debug logs: ✅

programmerq avatar Mar 29 '23 16:03 programmerq

workaround for lambda is to reference an s3 bucket instead.

The lambda api is limited to 50MB zipfiles on the AWS side.

See: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html#gettingstarted-package-awsother

GavinFrazar avatar Feb 15 '24 21:02 GavinFrazar

this was fixed in #40242

GavinFrazar avatar May 21 '24 08:05 GavinFrazar