bug: forward-auth with request_method GET causes timeouts

[BrandonArp]

Current Behavior

The request to the forward-auth request contains a Content-Length header and the GET request has no body.

Expected Behavior

The forward-auth request has no Content-Length header.

Error Logs

apisix-1  | 2024/02/07 20:27:39 [error] 242#242: *648462 [lua] forward-auth.lua:134: phase_func(): failed to process forward auth, err: timeout, client: 172.27.0.1, server: _, request: "POST /test HTTP/1.1", host: "localhost:9080"
apisix-1  | 2024/02/07 20:27:39 [warn] 242#242: *648462 [lua] plugin.lua:1159: run_plugin(): forward-auth exits with http status code 403, client: 172.27.0.1, server: _, request: "POST /test HTTP/1.1", host: "localhost:9080"

Steps to Reproduce

1. Run APISIX
2. Deploy a service that can consume forward-auth requests
3. Create a route that has forward-auth enabled, with a GET request_method You can use the following plugin config:

   forward-auth:
        keepalive: true
        keepalive_pool: 100
        keepalive_timeout: 2000
        request_headers:
          - Authorization
        request_method: GET
        uri: http://some_auth_service/auth```
   

4. Send 2 requests to the protected route
5. Observe that the second request will fail with a timeout

Environment

• APISIX version 3.8.0

[BrandonArp]

I believe that the issue was introduced in #10589 where some headers were added unconditionally here.

[moonming]

welcome PR to fix it @BrandonArp

[pshettyk]

I am also facing the same issue in 3.8.0. Works fine in 3.6.0.

[moonming]

@shreemaan-abhishek please take a look

[shreemaan-abhishek]

I have found the cause of this bug. I will fix this later.

Case 1:

• request contains payload
• original request method: GET
• fw auth request method: GET

In this case, APISIX unconditionally adds the Content-Length header to the request to be sent to fw-auth.uri but the request body (to fw-auth.uri) is empty. This causes the fw-auth server to wait endlessly to read the content.

Solution:

• add content-length header only if the fw-auth.request_method is POST.

Case 2:

• request contains payload
• original request method: POST
• fw auth request method: POST

In this case, APISIX passes client-body-reader by default. (this was added to support passing large request body to fw-auth). client-body-reader reads the request body and makes it empty. But since the content-length is non zero the upstream server again waits endlessly to read the request body which is no longer available.

Solution:

1. Use the client-body-reader only if the request body is too large. (How much is too large is unknown)
2. If using client-body-reader store the req-body in a var before sending the request to fw-auth server. After request from fw-auth is successful, restore the request body.

[suryaprabhakark]

Hi, I raised a PR for the fix. I faced the same issue and we are running the fixed code in our prod for sometime now, not seen any issues yet. Please let me know if i missed anything.

https://github.com/apache/apisix/pull/11021

[motongxue]

I have found the cause of this bug. I will fix this later.

Case 1:

• request contains payload
• original request method: GET
• fw auth request method: GET

In this case, APISIX unconditionally adds the Content-Length header to the request to be sent to fw-auth.uri but the request body (to fw-auth.uri) is empty. This causes the fw-auth server to wait endlessly to read the content.

Solution:

• add content-length header only if the fw-auth.request_method is POST.

Case 2:

• request contains payload
• original request method: POST
• fw auth request method: POST

In this case, APISIX passes client-body-reader by default. (this was added to support passing large request body to fw-auth). client-body-reader reads the request body and makes it empty. But since the content-length is non zero the upstream server again waits endlessly to read the request body which is no longer available.

Solution:

1. Use the client-body-reader only if the request body is too large. (How much is too large is unknown)
2. If using client-body-reader store the req-body in a var before sending the request to fw-auth server. After request from fw-auth is successful, restore the request body.

Hello abhishek. I have submitted a https://github.com/apache/apisix/pull/11023 regarding your issue, and I hope it can be of help to you. At the same time, I would greatly appreciate your assistance in reviewing and finalizing the test cases.

[OnGoingLzy]

I had the same problem with how to get the body to pass upstream correctly

[zhoujiexiong]

I had the same problem with how to get the body to pass upstream correctly

@OnGoingLzy FYI: #11350