Spring Cloud Gateway GRPC routing doesn't provide errors correctly

[rrevyakin]

It seems like Spring Cloud Gateway routing GRPC requests not properly - when server responds with error gateway provides it to client incorrectly which leads to error lose because client can't handle it.

I've got 3 apps - grpc-client, grpc-simple-gateway and grpc-server from Spring GRPC gateway example project (original article, original project on github). Algorythm is - grpc-client sends request to gateway, it routes request to grpc-server, grpc-server answers with a message to gateway, and gateway returns message to the grpc-client.

When grpc-server handle request with success (grpc-status = 0) then everything works fine - client correctly obtains the response. But when grpc-server responds with an exception (grpc-status != 0) then gateway 1) sends this error message to client, but for some reason not closes the stream 2) sends second empty message just to close the stream. Client skips first error message (because error message must close the stream) and read only empty last message. In clients logs it looks this way:

// Below receiving of message with error (grpc-status = 5 which means not-found) but for some reason stream is not closed: endStream = false
DEBUG 38436 --- [-worker-ELG-1-2] i.g.n.s.i.grpc.netty.NettyClientHandler  : [id: 0x66c97813, L:/127.0.0.1:54323 - R:localhost/127.0.0.1:8090] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[:status: 200, x-request-header: header-value, trailer: grpc-status, content-type: application/grpc, grpc-status: 5, grpc-message: test exception] padding=0 endStream=false
// Then gateway sends second message just to close a stream: endStream=true
DEBUG 38436 --- [-worker-ELG-1-2] i.g.n.s.i.grpc.netty.NettyClientHandler  : [id: 0x66c97813, L:/127.0.0.1:54323 - R:localhost/127.0.0.1:8090] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[grpc-status: 0] padding=0 endStream=true

It leads to error: INTERNAL: No value received for unary call - which is incorrect.

If omit gateway and invoke server directly from client then everything works fine - client receives only one message with end of stream and client happily handle it. In clients logs it looks this way:

// Only one message - with endStream = true
DEBUG 40800 --- [-worker-ELG-1-2] i.g.n.s.i.grpc.netty.NettyClientHandler  : [id: 0x354d0b37, L:/127.0.0.1:54349 - R:localhost/127.0.0.1:6565] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-status: 5, grpc-message: test exception] padding=0 endStream=true

It leads to error: NOT_FOUND: test exception - which is correct.

Steps to reproduce: I forked original spring example project, add unconditional response with error from grpc-server and enable trace logs - reproducible example based on original spring cloud gateway grpc example. Steps to reproduce are to just clone project and run this:

./gradlew :grpc-server:bootRun
./gradlew :grpc-simple-gateway:bootRun
./gradlew :grpc-client:bootRun

Meaningful logs with commentary are here in gist

Conclusion - it looks like gateway must send endStream=true with an error message but it doesn't.

Update

After long investigation we found that described bug seems like yet not implemented functionality. Below are short results of investigation:

When preparing a response in the netty implementation, the endStream http/2 flag is set - it depends on what type the nettyResponse variable will be in HttpServerOperations: endStream=true is set if nettyResponse instanceof LastHttpContent. After contacting the server, HttpClientResponse res contains the correct response. It can be DefaultFullHttpResponse (implements LastHttpContent) or DefaultHttpResponse (does not implement LastHttpContent). The problem is that when creating HttpServerOperations, the nettyResponse field is instantiated by DefaultHttpResponse (it is then populated with data from HttpClientResponse res). That is, even if HttpClientResponse was DefaultFullHttpResponse in the nettyResponse field, HttpServerOperations will still have DefaultHttpResponse, which leads to the fact that when sending a response from gateway to the client the response instanceOf LastHttpContent check is always false, which in turn leads to setting endStream=false. That is, in the process of preparing a response from the server to be sent to the client, endStream=true is always lost.

We also made a workaround solution which fixes this behaviour. It consists of two parts:

1. Main changes are made in reactory-netty, you can find it here (all in one commit): https://github.com/rrevyakin/reactor-netty-with-grpc-workaround/tree/v.1.0.21.1
2. Simple Spring Cloud GRPC Gateway with improved (hacked) NettyRoutingFilter: https://github.com/rrevyakin/spring-cloud-grpc-gateway-with-workaround

Solution is a hack but it can be useful for you in investigation of this bug.