spring-boot-protocol
spring-boot-protocol copied to clipboard
wangzihaogithub
FileUpload OutOfDirectMemoryError
当我在上传大文件的时候会发生OOM 下面是文件上传代码: `
@RequestMapping(value = "/stream-upload-test")
public ResponseEntity<String> upload(HttpServletRequest request, HttpServletResponse response)
throws IOException, FileUploadException {
boolean isMultipart = ServletFileUpload.isMultipartContent(request);
if (isMultipart) {
ServletFileUpload upload = new ServletFileUpload();
Map<String, String> params = new HashMap<>();
InputStream is = null;
FileItemIterator iter = upload.getItemIterator(request);
while (iter.hasNext()) {
FileItemStream item = iter.next();
if (!item.isFormField()) {
is = item.openStream();
try {
int i = 0;
byte bb [] = new byte[4096];
while((i = is.read(bb)) != -1){
System.out.println(Arrays.toString(bb));
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if (null != is) {
is.close();
}
}
} else {
String fieldName = item.getFieldName();
String value = Streams.asString(item.openStream());
System.out.println(fieldName + " : " + value);
}
}
}
return new ResponseEntity<String>(HttpStatus.OK);
}
`
错误信息截图如下：
解决了. 在这次提交, https://github.com/wangzihaogithub/spring-boot-protocol/commit/d65fe249948bb39ea437831850b1f4b27b081a1d 已经合并到master分支了, 也已经提交到maven中央仓库了. 过两天就同步到中央仓库了.
原因是: 错误的将netty的HttpPostMultipartRequestDecoder 的 discardThreshold参数设置成了Integer.MAX_VALUE导致的.
public int getDiscardThreshold(){ int discardThreshold = 0; if(multipartConfigElement != null) { discardThreshold = (int)multipartConfigElement.getMaxFileSize(); } if(discardThreshold <= 0){ discardThreshold = Integer.MAX_VALUE; //就是这里, 这个参数是控制复制文件的缓冲区, 我现在改成了不设置, 即netty默认的最大10M. } return discardThreshold; }
另外我在测试下载大文件的时候，耗时比tomcat长很多，从抓取的数据包发现，下载文件的时候发送一部分包后，会等待一段时间才会继续发送下一个包：
下面是我的测试代码： `@RequestMapping("/file-download-test")
public ResponseEntity<String> downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception {
String fileName = "CentOS-7-x86_64-DVD-2003.iso";
String filePath = "./" + fileName;
handleDownloadStream(fileName, filePath, request, response);
return new ResponseEntity<>(HttpStatus.OK);
}
public void handleDownloadStream(String fileName, String filePath, HttpServletRequest request, HttpServletResponse res) throws IOException { byte[] buffer = new byte[4 * 1024]; OutputStream os = null; FileInputStream fStream = null; try { os = new BufferedOutputStream(res.getOutputStream()); res.reset(); String agent = request.getHeader("User-Agent"); if (agent == null) { return; } agent = agent.toUpperCase();
//ie浏览器,火狐,Edge浏览器
if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
} else {
fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
}
//safari RFC 5987标准
if (agent.indexOf("SAFARI") > -1) {
res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
} else {
res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
}
File file = new File(filePath);
res.setContentType("application/octet-stream");
res.setCharacterEncoding("UTF-8");
res.setContentLength((int) file.length());
fStream = new FileInputStream(file);
int length = 0;
while ((length = fStream.read(buffer)) != -1) {
os.write(buffer, 0, length);
}
os.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(fStream);
}
}`
这是由这个参数控制的， 你可以改成1KB。默认达到是1M后，或者主动调用flush方法，再给客户端发数据。
server.netty.http-servlet.response-max-buffer-size=1024
我下个版本把默认值改小点，另外， 你可以这样发文件，性能会好一些，因为用的是netty的零拷贝
NettyOutputStream nettyOutputStream = ((NettyOutputStream)response.getOutputStream())； nettyOutputStream.write(new File("/home/temp.json"));
文件下载是否存在内存不释放的问题，配置启动内存-Xmx512M -XX:MaxDirectMemorySize=512M，下载一个不到2G文件会卡住，并且会将分配的内存耗尽。
可以看下代码吗？ 我用的你第一次的代码复现不出来。
master分支的这个类 com.github.netty.http.example.HttpController， 是我用的例子， 不知道一样不
`
@RequestMapping("/downloadFile") public ResponseEntity<String> downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception { String fileName = "CentOS-7-x86_64-DVD-2003.iso";
byte[] file = new byte[1024 * 1024 * 7];
for (int i = 0; i < file.length; i++) {
file[i] = (byte) i;
}
handleDownloadStream(fileName, new ByteArrayInputStream(file), request, response);
return new ResponseEntity<>(HttpStatus.OK);
}
public void handleDownloadStream(String fileName, InputStream inputStream, HttpServletRequest request, HttpServletResponse res) throws IOException {
byte[] buffer = new byte[4 * 1024];
OutputStream os = null;
try {
os = new BufferedOutputStream(res.getOutputStream());
res.reset();
String agent = request.getHeader("User-Agent");
if (agent == null) {
return;
}
agent = agent.toUpperCase();
//ie浏览器,火狐,Edge浏览器
if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
} else {
fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
}
//safari RFC 5987标准
if (agent.contains("SAFARI")) {
res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
} else {
res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
}
res.setContentType("application/octet-stream");
res.setCharacterEncoding("UTF-8");
res.setContentLength(inputStream.available());
int length = 0;
while ((length = inputStream.read(buffer)) != -1) {
os.write(buffer, 0, length);
}
os.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(inputStream);
}
}
`
`@RequestMapping("/file-download-test") public ResponseEntity<String> downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception { String fileName = "CentOS-7-x86_64-DVD-2003.iso"; String filePath = "./" + fileName; handleDownloadStream(fileName, filePath, request, response); return new ResponseEntity<>(HttpStatus.OK); }
public void handleDownloadStream(String fileName, String filePath, HttpServletRequest request, HttpServletResponse res) throws IOException {
byte[] buffer = new byte[4 * 1024];
OutputStream os = null;
FileInputStream fStream = null;
try {
os = new BufferedOutputStream(res.getOutputStream());
res.reset();
String agent = request.getHeader("User-Agent");
if (agent == null) {
return;
}
agent = agent.toUpperCase();
//ie浏览器,火狐,Edge浏览器
if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
} else {
fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
}
//safari RFC 5987标准
if (agent.indexOf("SAFARI") > -1) {
res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
} else {
res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
}
File file = new File(filePath);
res.setContentType("application/octet-stream");
res.setCharacterEncoding("UTF-8");
fStream = new FileInputStream(file);
int length = 0;
while ((length = fStream.read(buffer)) != -1) {
os.write(buffer, 0, length);
os.flush();
}
os.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
IOUtils.closeQuietly(fStream);
}
}`
被下载的文件大小大概4个G
jvm启动参数：
下载的文件信息：
jvm启动参数：
------------------ 原始邮件 ------------------ 发件人: "wangzihaogithub/spring-boot-protocol" <[email protected]>; 发送时间: 2020年12月5日(星期六) 下午4:19 收件人: "wangzihaogithub/spring-boot-protocol"<[email protected]>; 抄送: "tntym"<[email protected]>;"Author"<[email protected]>; 主题: Re: [wangzihaogithub/spring-boot-protocol] FileUpload OutOfDirectMemoryError (#12)
辛苦发一下吧 感谢
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
解决了， 这是netty的bug，已经修复了在master分支了。 后续pull-request 给netty。
修复netty的写失败后，没有再重试的bug。会导致下次无法再写入数据。
https://github.com/wangzihaogithub/spring-boot-protocol/commit/af43b8977adbb5dec65571369e9bd213ab8cc301
https://github.com/netty/netty/issues/10353
netty before
`
int attemptedBytes = buffer.remaining();
final int localWrittenBytes = ch.write(buffer);
if (localWrittenBytes <= 0) {
incompleteWrite(true);
return;
}
adjustMaxBytesPerGatheringWrite(attemptedBytes, localWrittenBytes, maxBytesPerGatheringWrite);
in.removeBytes(localWrittenBytes);
--writeSpinCount;
`
netty after
`
int attemptedBytes = buffer.remaining();
final int localWrittenBytes = ch.write(buffer);
if (localWrittenBytes <= 0) {
incompleteWrite(true);
}else {
adjustMaxBytesPerGatheringWrite(attemptedBytes, localWrittenBytes, maxBytesPerGatheringWrite);
in.removeBytes(localWrittenBytes);
}
--writeSpinCount;
`
使用commons-fileupload上传文件，以流的方式读取上传的数据，这个过程也是需要等上传的数据发送完成暂存在本地，才会到controller层处理么。直接读取流数据的方式上传文件，可不可以做到不在本地暂存一份。
可以， 我改下。 不过这种模式和servlet的模式冲突(因为servlet要求解析)， 我打算加个配置参数，让你可以选择开启或关闭。
这个配置可以通过 重写spring提供CommonsMultipartResolver的来处理么： `public class CommonsMultipartResolverForProgress extends CommonsMultipartResolver {
@Override
public boolean isMultipart(HttpServletRequest request) {
if (request.getRequestURI().startsWith("/stream-upload-test")) {
return false;
}
return super.isMultipart(request);
}
} `
改好了，在master分支， 你可以重写spring提供CommonsMultipartResolver的来处理了。 这是这次改动， https://github.com/wangzihaogithub/spring-boot-protocol/commit/603ce3a4159095f6d63b26480145c01cd001fd11
如果你需要在maven中央仓库中， 需要我发新版本，可以直接告诉我，我就发新版了。