how to send requests with large body

[shreemaan-abhishek]

I am facing problems sending requests with large body as it fails with a connection refused error. Here is the code snippet that doesn't work:

local http = require("resty.http")

local tempFileName = os.tmpname()
local file = io.open(tempFileName, "wb")

local fileSizeInBytes = 11 * 1024 * 1024 -- 11MB
for i = 1, fileSizeInBytes do
    file:write(string.char(0))
end
local file = io.open(tempFileName, "rb")
local large_body = file:read("*all")
file:close()

local uri = "http://127.0.0.1:9980/lol"

local httpc = http.new()
local res1, err = httpc:request_uri(uri,
    {
        method = "POST",
        body = large_body
    }
)
if err then
  print(err)
end
print(res1.status)

Here is a capture of the request using wireshark: https://drive.google.com/file/d/19UajGULTjN4eJJKfSKdzHV7JwpwxEPWm/view?usp=sharing

[lasunpyaaj]

@pintsized any insights on this? I am also facing the same issue.

[pintsized]

What does "doesn't work" mean? I haven't got time to analyse wireshark output for you, if you need help then please provide more detail.

[shreemaan-abhishek]

sorry about that. It gives a connection refused error.

That said, is there any limitation imposed by the library? The code I shared works absolutely fine if I remove the large body or use a smaller body.

[pintsized]

Before any request is made, a connection must be established. This error is telling you that, for some reason, your Nginx server cannot open a connection to the URI. You'll want to investigate that.

[shreemaan-abhishek]

I just figured it out that using client_body_reader as the request body for large request body works just fine. Also, replacing the large request body with a smaller one for the exact same code also works fine.

My use case is a forward-auth application, where I send the request body to the auth server first and then continue sending the request to the upstream if auth passes. Using client_body_reader empties the request body for some reason.

So I wanted to know if I should do something extra when dealing with large request body? Is there a threshold, I could do something like

if request body is large
    body = httpc:get_client_body_reader
else
    body = ngx.req.get_body_data

Here is the original issue if you are interested

your Nginx server cannot open a connection to the URI.

The URI is perfectly fine, I can assure you of that.

[pintsized]

I can't say why you're having problems sending a largish body directly - there are no hardcoded limitations (other that nginx worker memory). If you really want to allocate 11MB of data in a string and push it over the wire in one go, well I guess it should work. But you probably don't want to do that.

The client_body_reader technique is for streaming the request body - in other words, in cases where you do not want to allocate the entire request body in memory, and want to deliver it in predictable chunk sizes in order to avoid large memory allocations. So yes, it won't be available as a string afterwards, because that's what you're trying to avoid by using streaming.

If you get a connection refused error at any point, this is literally the underlying TCP stack telling you that the connection is refused. In other words, that's not something happening in this module.

[yang-dang]

@shreemaan-abhishek Is there any progress on this issue?

[shreemaan-abhishek]

yes, sorry there was some problem with the upstream I was using.