req icon indicating copy to clipboard operation
req copied to clipboard

Multipart form upload use huge amount of memory

Open n0v3xx opened this issue 2 years ago • 3 comments

Hi, i think there is a bug or something in your implementation, because if i want to upload a file here against this API: https://rapidgator.net/article/api/file#upload

v3 - bad memory usage way: I tried all upload ways, SetFileReader, SetFiles and so on

ursp2 := &UploadResponse{}
resp, err := r.Client.Client.R().
	SetHeader("User-Agent", Useragent).
	SetFileReader("file", fileName, rr).
	//SetFiles(map[string]string{
	//	"file": filePath,
	//}).
	SetSuccessResult(ursp2).
	SetErrorResult(&errMsg).
	Post(ursp.Response.Upload.URL)
if err != nil {
	mursp.Error = err
	r.Logger.ErrorLogger.Println(err.Error())
	fmt.Println(err)
	//c <- mursp
	return
}

The upload process load the hole file in memory and that use tons of RAM. I dont know why but i implement a plain multipart file upload like this here, that works fine, without this memory bug.

Could you try to improve your multipart form request?

Your old v0.3.2 works. No extrem RAM usage

n0v3xx avatar Apr 16 '23 12:04 n0v3xx

I don't think it's possible, SetFileReader and SetFiles will not load the hole file in memory, just copy from reader to server, I'll check it out in a few days when I'm free.

imroc avatar Apr 16 '23 12:04 imroc

I dont know why but the hole file is loaded to RAM but only for this filehoster. I tested other ones it works with your client. And i noticed that my "working solution" also not work... i dont understand why that happens. If you want i can send you test credentials for this filehoster and the code i use. If you want i send it to you via mail.

n0v3xx avatar Apr 17 '23 09:04 n0v3xx

I have done some more tests. Your old version v0.3.2 works like a charm, no RAM consumption at all. Your new version does not work. It loads the entire file into RAM. I don't know what you changed, but this is not good.

n0v3xx avatar May 13 '23 20:05 n0v3xx