[core] LambdaRuntimeClient do nothing when server closes the connection

[sebsto]

Expected behavior

When the server closes the connection, the client should try to reconnect and eventually shutdown if it can't.

We should implement a retry strategy to re-establish the connection (ideally with an exponential backoff) and gracefully shut down the client if it can't reconnect after n attempts.

Actual behavior

The LambdaRunTimeClient change its internal state but does not attempt to reconnect nor exit. See this line of code https://github.com/swift-server/swift-aws-lambda-runtime/blob/b4f016422ebd5e9c2e6aa97b8c74e924dad2f4c9/Sources/AWSLambdaRuntimeCore/LambdaRuntimeClient.swift#L267

Steps to reproduce

1. checkout this branch to use the new MockServer https://github.com/sebsto/swift-aws-lambda-runtime/tree/sebsto/mockserver

2. Start the Mock server

LOG_LEVEL=trace  swift run

2025-01-15T11:43:07+0100 info MockServer : host="127.0.0.1" maxInvocations=1 port=7000 [MockServer] Server started and listening

3. From another terminal, start the HelloWorld example in DEBUG mode.

cd Examples/HelloWorld
export AWS_LAMBDA_RUNTIME_API=127.0.0.1:7000
LAMBDA_USE_LOCAL_DEPS=../.. LOG_LEVEL=trace swift run

uild of product 'MyLambda' complete! (0.12s)
2025-01-15T11:52:18+0100 debug LambdaRuntime : [AWSLambdaRuntimeCore] LambdaRuntime initialized
2025-01-15T11:52:18+0100 trace LambdaRuntime : lambda_ip=127.0.0.1 lambda_port=7000 [AWSLambdaRuntimeCore] Connection to control plane created

[hangs after the /next and /response requests have been processed and the mock server shutdown]

The mock server closes the connection after 2 HTTP request on 1 connection. The runtime will send the /next and the /response requests and then it will stay up and running without retrying the connection.

Adding a print() statement at line 267 shows that the connection close even is trapped, but no action is taken.

If possible, minimal yet complete reproducer code (or URL to code)

n/a

What version of this project (swift-aws-lambda-runtime) are you using?

main

Swift version

swift-driver version: 1.115.1 Apple Swift version 6.0.3 (swiftlang-6.0.3.1.10 clang-1600.0.30.1) Target: arm64-apple-macosx15.0

Amazon Linux 2 docker image version

n/a

[sebsto]

Having given a second tough about it, maybe we can just be a bad citizen and crash when the connection is closed. The Lambda service will recreate another microVM to continue to server client request.

The current performance testing script measures performance by calculating the runtime of an executable target. If we implement a retry with backoff strategy, this will impact the performance measurement (or we will need to redesign the perf testing)

Furthermore, the retry logic will add a few Kb to the library size that will be only needed in few rare occurence of the service stopping serving requests to the LambdaRuntime.

Maybe a simple fatalError() would be good enough in this situation.

Wdyt ?

[sebsto]

@fabianfett we have to re-evaluate this one after all the changes on the LambdaRuntimeClient

[sebsto]

Changes since I opened this issue.

1. The MockServer has been merged. You can run it with swift run MockServer
2. the new LambdaRuntimeClient crashes with

2025-07-04T17:16:52+0200 trace LambdaRuntime : [AWSLambdaRuntime] Close lambda runtime client
Swift/ErrorType.swift:254: Fatal error: Error raised at top level: AWSLambdaRuntime.LambdaRuntimeError(code: AWSLambdaRuntime.LambdaRuntimeError.Code.invocationMissingMetadata, underlying: Optional(AWSLambdaRuntime.LambdaRuntimeError(code: AWSLambdaRuntime.LambdaRuntimeError.Code.nextInvocationMissingHeaderRequestID, underlying: nil)))

I traced it down to this line of code https://github.com/swift-server/swift-aws-lambda-runtime/blob/4a7d95e4a38a6936074a0a7a3721fd0eaf4c666d/Sources/AWSLambdaRuntime/LambdaRuntimeClient.swift#L149