Space in query parameter is sent to gRPC backend as plus.

[dseomn]

I just sent an HTTP request with the query parameter filter=%2b%20 (plus then space), and the gRPC backend saw the proto field as filter: "++". The Kubernetes manifest shows that we're using gcr.io/endpoints-release/endpoints-runtime@sha256:793558dd6250da969b9108cc0615f4dfd0796cc903d2b5c865014eabfc59f912, which appears to be ESP 2.25.0.

[qiwzhang]

Unescape code is here. We don't see any problem with it. It has been unit-tested here

Could you turn on --enable_debug in ESP startup arg, and get its log for us.

[dseomn]

The log has a bunch of stuff that I'm not sure is a good idea to share publicly, but I tried to pick out what are hopefully the useful bits. I can send the full log to your @google.com email if that would help, or I could share my screen over Meet.

Is the first line below showing what's coming in to ESPv2, and the second line showing what it's sending to the backend? Is there any way to see the body of the POST it made to the backend?

D0318 22:14:45.387 31 envoy] [31][http][C6][S<redacted>] request headers complete (end_stream=true):\n\':method\', \'GET\'\n\':scheme\', \'http\'\n\':path\', \'/<redacted>?filter=%2B+&pageSize=10\'<redacted ...>
...
D0318 22:14:45.388 31 envoy] [31][router][C6][S<redacted>] router decoding headers:\n\':method\', \'POST\'\n\':scheme\', \'http\'\n\':path\', \'<redacted>\'\n\':authority\', \'<redacted>\'\n\'x-request-id\', \'<redacted>\'\n\'x-real-ip\', \'<redacted>\'\n\'x-forwarded-for\', \'<redacted>\'\n\'x-forwarded-host\', \'<redacted>\'\n\'x-forwarded-port\', \'443\'\n\'x-forwarded-proto\', \'https\'\n\'x-original-uri\', \'<redacted>?filter=%2B+&pageSize=10\'\n\'x-scheme\', \'https\'\n\'user-agent\', \'python-requests/2.22.0\'\n\'accept-encoding\', \'gzip, deflate\'\n\'accept\', \'application/json\'\n\'authorization\', \'<redacted>'\n\'x-endpoint-api-userinfo\', \'<redacted>'\n\'content-type\', \'application/grpc\'\n\'x-envoy-original-path\', \'<redacted>?filter=%2B+&pageSize=10\'\n\'x-envoy-original-method\', \'GET\'\n\'te\', \'trailers\'\n\'x-envoy-expected-rq-timeout-ms\', \'15000\'\n\'traceparent\', \'<redacted>\'\n\'x-cloud-trace-context\', \'<redacted>\'

[nareddyt]

Hi @dseomn, you can email [email protected] and [email protected] the full log.

Yes:

• [http]... request headers complete is the request headers received from the client.
• [router]... router decoding headers is the request headers to the backend.

Can you double check what query parameter you sent ESPv2? You mentioned filter=%2b%20, but the logs above show filter=%2B+. Do you have something else in front of ESPv2 that is modifying it?

Unfortunately you cannot view the request body in the log, but it will still be useful for us to check. I will try reproducing the issue myself.

[nareddyt]

This is interesting. It seems someone else has the same issue: https://github.com/envoyproxy/envoy/issues/13965

[dseomn]

Thanks, I just sent both of you the full logs.

I made the request with filter=%2b%20&pageSize=10, but we have nginx and I think some sort of load balancer between the curl command I ran and ESPv2. I'm guessing nginx converted %2b%20 to %2B+ (the two are equivalent).