Uploading large files currently causes the entire file to be buffered in memory

[yashgorana]

repro steps

1. Replace [PATH] with a 100MB+ file.
2. observe memory usage

package main

import (
	"fmt"

	"github.com/imroc/req/v3"
)

func main() {
	client := req.C().EnableDumpAllWithoutBody()

	res, err := client.R().
		SetFile("file", "[PATH]").
		Put("https://httpbin.org/put")

	fmt.Printf("res: %v\n", res)
	fmt.Printf("err: %v\n", err)
}

Example 1

for reference, I was using 146MB Zotero-7.0.15.dmg which resulted in 330MB utilization (2x)

:authority: httpbin.org
:method: PUT
:path: /put
:scheme: https
content-type: multipart/form-data; boundary=c8f0f7e35932977c02dee27370c6e12a48f10bb39b78564277443dcdfddc
content-length: 146829893
accept-encoding: gzip
user-agent: req/v3 (https://github.com/imroc/req)

Example 2

When uploading a 670MB Video it eats up 1.7GB.

:authority: httpbin.org
:method: PUT
:path: /put
:scheme: https
content-type: multipart/form-data; boundary=eeff6d9b0cef67c88ad5a543a4957a3707d9faf041db10ed62ef09e34feb
content-length: 672755483
accept-encoding: gzip
user-agent: req/v3 (https://github.com/imroc/req)

[yashgorana]

Looks like this code path triggers the buffering when pw = w.CreatePart(...) https://github.com/imroc/req/blob/master/middleware.go#L122-L123

[ManuelReschke]

I think that is the same issue here: https://github.com/imroc/req/issues/231