serverless-java-container Spring Cloud Gateway throwing "hostname can't be null" based on my request only having headers and not multi-value headers

Serverless Java Container version: com.amazonaws.serverless:aws-serverless-java-container-springboot3:2.0.0

Implementations: Spring Cloud gateway

Framework version: org.springframework.cloud:spring-cloud-starter-gateway:4.1.1

Frontend service: ALB

Deployment method: localstack

Scenario

I am attempting to run Spring Cloud Gateway in a lambda environment, using AWS's Serverless Java container for Spring Boot 3. I am deploying with localstack locally and hitting it with requests.

I have a /health endpoint:

  @Bean
  public RouterFunction<ServerResponse> health() {
    return RouterFunctions.route(
        RequestPredicates.path("/health"), request -> ServerResponse.ok().build());
  }

which works as expected (I get a 200 OK back).

However, hitting one of my gateway endpoints I get the following

I have a pre-request filter saying I matched to my gateway configuration so it is forwarding a request
I have a post-request filter saying it called the downstream service (ie the proxy worked) and it returned 200
I then get a "hostname can't be null" exception from InetSocketAddress

I've tracked this down to:

a header filter tries to get the 'remote address'
this calls upon the HttpServletRequest for a host
which hits AwsProxyHttpServletRequest's AwsProxyRequest asking for a multi-value header describing host

However, using debugging I can see I only have headers rather than multi-value headers: meaning .getHeaders().getFloorEntry("Host").getValue() is more likely what I need.

Does anyone here have advice on how to work the problem?

Expected behavior

Proxy requests should work and return my request.

Actual behavior

There is an exception when looking up a hostname due to it being null, based on the request I am sending into my service.

Steps to reproduce

n/a, but the app is currently very vanilla (so I don't think there is anything getting in the way). I have 3 beans, 1 for health (above) and 2 that do the pre/post proxy request logging. Nothing else is present in the service.

Full log output

2024-03-26 16:27:40 2024-03-26T16:27:40.405Z  INFO 17 --- [           main] c.a.s.p.internal.LambdaContainerHandler  : 127.0.0.1 null- null [26/03/2024:16:27:40Z] "GET /health null" 200 - "-" "-" combined
2024-03-26 16:27:40 2024-03-26T16:27:40.965Z  INFO 17 --- [           main] u.c.c.s.t.a.f.LoggingGlobalPreFilter     : Matched route id issuing_account_status: Forwarding request /account/{accountId}/status to http://wiremock:9090/account/123/status
2024-03-26 16:27:40 2024-03-26T16:27:40.967Z  INFO 17 --- [           main] u.c.c.s.t.a.f.LoggingGlobalPostFilter    : Response from http://wiremock:9090/account/123/status: 200 OK
2024-03-26 16:27:40 2024-03-26T16:27:40.975Z ERROR 17 --- [           main] a.w.r.e.AbstractErrorWebExceptionHandler : [3a588b5f]  500 Server Error for HTTP GET "/account/123/status"
2024-03-26 16:27:40
2024-03-26 16:27:40 java.lang.IllegalArgumentException: hostname can't be null
2024-03-26 16:27:40     at java.base/java.net.InetSocketAddress.checkHost(Unknown Source) ~[na:na]
2024-03-26 16:27:40     Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
2024-03-26 16:27:40 Error has been observed at the following site(s):
2024-03-26 16:27:40     *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
2024-03-26 16:27:40     *__checkpoint ⇢ HTTP GET "/issuing/account/123/status" [ExceptionHandlingWebHandler]
2024-03-26 16:27:40 Original Stack Trace:
2024-03-26 16:27:40             at java.base/java.net.InetSocketAddress.checkHost(Unknown Source) ~[na:na]
2024-03-26 16:27:40             at java.base/java.net.InetSocketAddress.<init>(Unknown Source) ~[na:na]
2024-03-26 16:27:40             at org.springframework.http.server.reactive.ServletServerHttpRequest.getRemoteAddress(ServletServerHttpRequest.java:192) ~[spring-web-6.1.1.jar:6.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.filter.headers.ForwardedHeadersFilter.filter(ForwardedHeadersFilter.java:116) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.filter.headers.HttpHeadersFilter.filter(HttpHeadersFilter.java:38) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.filter.headers.HttpHeadersFilter.filterRequest(HttpHeadersFilter.java:28) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.filter.NettyRoutingFilter.filter(NettyRoutingFilter.java:125) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:137) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:44) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:117) ~[spring-cloud-gateway-server-4.1.1.jar:4.1.1]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:264) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:264) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:258) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:863) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onNext(MonoFilterWhen.java:136) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.request(MonoFilterWhen.java:182) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:338) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:338) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:264) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at reactor.core.publisher.Mono.subscribe(Mono.java:4512) ~[reactor-core-3.6.0.jar:3.6.0]
2024-03-26 16:27:40             at org.springframework.http.server.reactive.ServletHttpHandlerAdapter.service(ServletHttpHandlerAdapter.java:199) ~[spring-web-6.1.1.jar:6.1.1]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.spring.embedded.ServerlessReactiveServletEmbeddedServerFactory.service(ServerlessReactiveServletEmbeddedServerFactory.java:71) ~[aws-serverless-java-container-springboot3-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.internal.servlet.FilterChainManager$ServletExecutionFilter.doFilter(FilterChainManager.java:374) ~[aws-serverless-java-container-core-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.internal.servlet.FilterChainHolder.doFilter(FilterChainHolder.java:90) ~[aws-serverless-java-container-core-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.internal.servlet.AwsLambdaServletContainerHandler.doFilter(AwsLambdaServletContainerHandler.java:154) ~[aws-serverless-java-container-core-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:183) ~[aws-serverless-java-container-springboot3-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.spring.SpringBootLambdaContainerHandler.handleRequest(SpringBootLambdaContainerHandler.java:56) ~[aws-serverless-java-container-springboot3-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:215) ~[aws-serverless-java-container-core-2.0.0.jar:na]
2024-03-26 16:27:40             at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxyStream(LambdaContainerHandler.java:258) ~[aws-serverless-java-container-core-2.0.0.jar:na]
2024-03-26 16:27:40             at my.LambdaHandler.handleRequest(LambdaHandler.java:36) ~[task/:na]
2024-03-26 16:27:40             at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$2.call(EventHandlerLoader.java:905) ~[aws-lambda-java-runtime-interface-client-2.4.2-linux-x86_64.jar:2.4.2]
2024-03-26 16:27:40             at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:245) ~[aws-lambda-java-runtime-interface-client-2.4.2-linux-x86_64.jar:2.4.2]
2024-03-26 16:27:40             at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:197) ~[aws-lambda-java-runtime-interface-client-2.4.2-linux-x86_64.jar:2.4.2]
2024-03-26 16:27:40             at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.main(AWSLambda.java:187) ~[aws-lambda-java-runtime-interface-client-2.4.2-linux-x86_64.jar:2.4.2]
2024-03-26 16:27:40
2024-03-26 16:27:40 2024-03-26T16:27:40.995Z  INFO 17 --- [           main] c.a.s.p.internal.LambdaContainerHandler  : 127.0.0.1 null- null [26/03/2024:16:27:40Z] "GET /account/123/status null" 500 150 "-" "-" combined

Mar 27 '24 09:03 tc2-c1

Update: I think the problem is that I interpreted that out-of-the-box this supports ALB messages (as well as API GW), but reading the AwsProxyRequest javadoc I don't think it does 😞

Mar 27 '24 16:03 tc2-c1

Thank you for raising this @tc2-c1 . If the Hostname only appears on singlevalueHeaders for ALB, then we should update the method to account for that. I will send a fix by the end of the day. Thanks again

Mar 27 '24 17:03 mbfreder

Thanks @mbfreder , the response is massively appreciated 🙏

I have noticed that this line: return new InetSocketAddress(this.request.getRemoteHost(), this.request.getRemotePort()); implies that there are different headers for host and port, but the header that has come through in my screenshot above is host:port in just the 1 field, so this may need to be split...

Mar 27 '24 17:03 tc2-c1

Released as part of 2.0.2

May 31 '24 08:05 deki

serverless-java-container serverless-java-container copied to clipboard

Spring Cloud Gateway throwing "hostname can't be null" based on my request only having headers and not multi-value headers

Scenario

Expected behavior

Actual behavior

Steps to reproduce

Full log output

serverless-java-container
serverless-java-container copied to clipboard