spring-cloud-gateway
spring-cloud-gateway copied to clipboard
spring-cloud
ProxyExchange fails with "IOException: insufficient data written" with boot 3.2
The ProxyExchange type in spring-cloud-gateway-mvc copies over the headers from the incoming request, unless they're marked as sensitive. In addition, it also uses the HTTP message converters to unmarshal the payload of the incoming request and marshal it again when sending.
What we found is that in the header copying, it also includes the Content-Length header of the incoming request. However, b/o the unmarshalling and then marshalling the content length of the proxied request can be different: especially in the case of JSON where whitespace is removed, it can become less.
This causes problems with some downstream servers, as they keep waiting for additional request contents based on the erroneous Content-Length header. When not using smth like Apache Component HTTP client but using the default Java client instead, this even leads to errors in the client itself (showcased in attached sample).
This behavior was not there in previous versions: actually, with Spring Boot 3.1.6 and the underlying Spring web version this works. When you check what's being sent then you can see that a Content-Length header is always set, with the correct value.
It looks like the latest Spring web code (Jackson HTTP message converter?) no longer sets its own Content-Length header when marshalling to JSON. Therefore, spring-cloud-gateway-mvc should never copy over this header from the incoming request, but should rather leave it up to the RestTemplate to either leave it out or set this to the correct value for the outgoing request.
BTW, although this isn't strictly related to this issue, I believe that the Host header of the incoming request shouldn't be copied either by default: this can also lead to issues.
Sample
I've attached a project with two integration tests that showcase the problem. By configuring the content-length to be a sensitive header you can work around the issue, but users of the framework should not be required to do this.
gateway-mvc-contentlength.zip
Posting for those encountering, the workaround is to set
spring.cloud.gateway.proxy.sensitive=content-length
The body isn’t modified by my code, why would I pass it myself? This is done automatically.
The way that we use this, we just want to pass the request with some modified headers and path to the downstream service, which I think is the typical use case for a proxy solution. The controller doesn't care about the type of a potential payload of the request, nor about the response payload type: that's why we use ProxyExchange<byte[]>.
We could explicitly add an @RequestBody byte[] body parameter to the controller to force the conversion to not use the Jackson message converter, but to me that feels like just another workaround: I think it's confusing to explicitly ask for input that you don't actually process in your controller method just to prevent the ProxyExchange from producing an invalid HTTP request (invalid in the sense that its Content-Length value is too low).
In our current use case we don't really mind the fact that the JSON message conversion is done: it also provides an additional layer of validation to ensure that the payload is in fact valid JSON. In general, since this is all just Spring MVC, other uses might involve unmarshalling the request body to some type (even outside of the controller) without the proxying code being interested in the body, and you'd expect that to work without having to compensate for headers with invalid values I think.
I am trying to work around this issue and I am not sure exactly how the property should look like:
spring.cloud.gateway.proxy.sensitive=content-lengthspring.cloud.gateway.mvc.proxy.sensitive=content-length
It seems like for routes are spring.cloud.gateway.mvc.routes....
Any clue about that? Btw, I am not using any Java code, just the configuration in the application.yml file.
Besides, the link to the current documentation is broken https://docs.spring.io/spring-cloud-gateway/reference/index.html
@estigma88 docs are back. This is only for using spring-cloud-starter-gateway-mvc not either of the server variants that have routes, so your issue is unrelated. The correct property is in my previous comment, but doesn't apply to your situation.
It might be related, this is an example of what's happening, the body is not passing through:
- From the URL /other/transcoder/v1/break
- To the URL /base/transcoder/v1/break
Request is passthrough but no body.
@estigma88 please open a separate issue. You code uses the gateway server mvc, this issue is around ProxyExchange, not server
Would it help to create a pull request to address this? I didn't consider that when I reported the issue as it seems trivial to fix, but given that it's been open for 4 months already and that various people have encountered issues b/o this bug I think it would be good to have a released solution ASAP.
OK, I've created https://github.com/spring-cloud/spring-cloud-gateway/pull/3313 to address the issue for both the MVC and Webflux version.
I thought the idea was that a PR could be considered for this week's release, but I see that the release was already cut without any feedback on the PR that I created last week... Any chance to get this into the next release, at least? Or any feedback on what might still be needed for that to happen?
Posting for those encountering, the workaround is to set
spring.cloud.gateway.proxy.sensitive=content-length
Note that setting it like this removes all the default sensitive headers and therefore is likely to let you expose secrets to downstream services that you’re proxying, so please don’t copy this as-is.
Hey, it's been 5 five months since reporting this and four weeks since providing a suggested fix: any chance of getting a review on the PR? Right now both ProxyExchange modules are broken as shipped in the latest releases...
Missed yet another release, I see. I'm going to give up on chasing this: it's very puzzling to me how an official Spring project simply ignores shipping two broken modules for half a year already when it's clear what the problem is and there's a proposed solution.
I just come here to say that I am also in that position. I upgraded spring boot 2.2.x to 3.2.x and this ProxyExchange is the simplest way for us to replace netflix-zuul.
But because of this header problem, for which a fix is waiting to be merged for many months now (thank you @jkuipers for providing that!), this migration is in limbo again.
It would be really nice to either put it into the next release or otherwise update us if there is a problem with that proposed solution (maybe the different configuration property is? Then please say so @spencergibb )
@spencergibb hello!
Can you download updated artifact (v 4.1.5) in mvnrepo please?