go-zero icon indicating copy to clipboard operation
go-zero copied to clipboard

fix: fix large file upload memory problem

Open zhaolunxiang123 opened this issue 2 years ago • 5 comments

fix large file upload memory problem

Log: fix large file upload memory problem

zhaolunxiang123 avatar Jun 14 '22 02:06 zhaolunxiang123

Don't do it in this complicated method, if it doesn't work for large files, better write another method to do it.

kevwan avatar Jun 21 '22 11:06 kevwan

Don't do it in this complicated method, if it doesn't work for large files, better write another method to do it.

The verbose parameter already exists in RestConf. Can I reuse it?When large files output logs, use the verbose parameter to control whether to copy the file stream using the new method @kevwan

zhaolunxiang123 avatar Jun 22 '22 01:06 zhaolunxiang123

Don't do it in this complicated method, if it doesn't work for large files, better write another method to do it.

The verbose parameter already exists in RestConf. Can I reuse it?When large files output logs, use the verbose parameter to control whether to copy the file stream using the new method @kevwan

I don't get your question, sorry!

kevwan avatar Jun 22 '22 13:06 kevwan

Don't do it in this complicated method, if it doesn't work for large files, better write another method to do it.

The verbose parameter already exists in RestConf. Can I reuse it?When large files output logs, use the verbose parameter to control whether to copy the file stream using the new method @kevwan

I don't get your question, sorry!

add new method DupReadCloserForLargeFile to do large files upload

zhaolunxiang123 avatar Jun 23 '22 02:06 zhaolunxiang123

When I was uploading large files, I found that when the middleware getLogHandler processed stream files, it read all streams into bytes.Buffer, which resulted in high memory. After checking, I found that the memory applied to the heap could not be released in time. For my PR, when setting the configuration file parameter 'Verbose=true', the file stream will be stored in temporary space to solve the problem of high memory @kevwan

zhaolunxiang123 avatar Aug 10 '22 02:08 zhaolunxiang123