azure-sdk-for-java icon indicating copy to clipboard operation
azure-sdk-for-java copied to clipboard
verified publisher icon Azure

[BUG] Java Heap OutOfMemoryError after library upgrade

Open AnastasiaBlack opened this issue 3 years ago • 8 comments

Describe the bug Hello, dear Azure Team! We have faced an interesting memory issue when upgrading the storage blob library. We used the older library azure-mgmt-storage 1.41.3 version and file upload for streams worked as expected. After upgrading the library to azure-storage-sdk 12.18.0 and using upload(dataStream, contentLength, true) method to stream file to Azure storage we started getting java.lang.OutOfMemoryError: Java heap space error. The error disappears after downgrading to the old version.

We also tried to use ParallelTransferOptions and upload the stream by smaller chunks (2 MB): long blockSize = 2 * Constants.MB; ParallelTransferOptions parallelTransferOptions = new ParallelTransferOptions() .setBlockSizeLong(blockSize) .setMaxConcurrency(2); blobClient.uploadWithResponse(inputStream, contentSize, parallelTransferOptions, null, null, null, new BlobRequestConditions(), null, Context.NONE);

It slightly improved the situation, but the error still occurred when uploading several files. The thing is that with the older version the error doesn't happen at all for the same files. It seems that SDK doesn't release the memory for some reason. This is why we have to keep the old version now because it works as expected, but it would be nice to be able to use the new version. Could you please investigate this problem? Thank you very much in advance for taking a look into this issue!

Exception or Stack Trace java.lang.OutOfMemoryError: Java heap space error

To Reproduce Steps to reproduce the behavior:

  1. Upgrade the library from azure-mgmt-storage 1.41.3 to azure-storage-sdk 12.18.0
  2. Upload a file (any size, we tried the 45 Mb and 67 Mb file)

Code Snippet

  1. upload(dataStream, file.length(), true) 

  2.          long blockSize = 2 * Constants.MB;
      ParallelTransferOptions parallelTransferOptions = new ParallelTransferOptions()
              .setBlockSizeLong(blockSize)
              .setMaxConcurrency(2);
      blobClient.uploadWithResponse(inputStream, contentSize, parallelTransferOptions, null, null,
              null, new BlobRequestConditions(), null, Context.NONE);

Expected behavior The file can be streamed to Azure storage without out of memory error

Setup (please complete the following information):

  • Library/Libraries: azure-storage-sdk 12.18.0
  • Java version: 17

AnastasiaBlack avatar Aug 29 '22 16:08 AnastasiaBlack

Hi, @AnastasiaBlack Thank you for reporting this issue. A few questions for you:

  • What is the heap size on your jvm? It looks like the files you're trying to upload are less than 100mb, which is pretty small to be causing an OOM.
  • Without the configuration changes to ParallelTransferOptions, does this happen even on a single file?
  • Does this reproduce every time?

Thanks! Rick

FYI: @jaschrep-msft @ibrahimrabab

rickle-msft avatar Aug 29 '22 18:08 rickle-msft

Hello @rickle-msft , thank you for your reply!

  • our jvm max heap size is 209.7 Mb
  • for upload(dataStream, file.length(), true) it happens even on a single file for ParallelTransferOptions it happens on multiple files only
  • before we upgraded to this new version form the old one, we never experienced this OOM, and we haven't changed the JVM heap size Thank you for your help!

AnastasiaBlack avatar Sep 01 '22 13:09 AnastasiaBlack

Hi @AnastasiaBlack ,

We tried to reproduce the issue you were seeing but were unable to see the OOM. Could you please share a code sample of what you are doing exactly, so that we could reproduce it on our end?

Thank you!

ibrahimrabab avatar Sep 06 '22 20:09 ibrahimrabab

Thank you very much for your reply! yes, sure! So before upgrade, the working code without any OOM looked like this (with old com.microsoft.azure:azure-storage:6.1.0 and azure-mgmt-storage 1.41.3)

CloudBlockBlob imgBlob = container.getBlockBlobReference(blob.getFilePath());
imgBlob.upload(inputStream, contentSize);

After updating to azure-storage-sdk 12.18.0 here Is the code getting OOM:

BlobClient blobClient = blobContainerClient.getBlobClient(blob.getFilePath());
blobClient.upload(inputStream, contentSize, true);

Could it be that the new lib azure-storage-sdk 12.18.0 changed the logic of streaming under the hood and this newer version buffers huge sizes of files compared to the old one?

AnastasiaBlack avatar Sep 07 '22 00:09 AnastasiaBlack

@AnastasiaBlack That is correct! The new lib has a different logic of streaming and buffers large chunks.

If possible, can you run a memory profiler on your application, and share information (and a screenshot) on what happened when the OOM occurs?

Also, based on the code snippet above, I don't see the ParallelTransferOptions being passed in. Have you tried passing in the ParallelTransferOptions? Another suggestion is also to set maxSingleUploadSize field within ParallelTransferOptions. This will allow you to upload a certain amount in one sitting. Let me know if these steps helped! If not, we can definitely take a look at what the memory profiler returns.

ibrahimrabab avatar Sep 07 '22 19:09 ibrahimrabab

I want to contribute in solving this issue. Can you please provide the approval ?

2021H1030044G avatar Sep 08 '22 14:09 2021H1030044G

You may create a PR for us to review. Please link it in this thread if you have a solution.

ibrahimrabab avatar Sep 08 '22 18:09 ibrahimrabab

Hi @AnastasiaBlack ,

Just following up on this thread. Has your issue been resolved? If so, we may go ahead and close this thread. If not, can you please let us know what blockers you are running into, or questions you may have?

Thank you!

ibrahimrabab avatar Sep 20 '22 21:09 ibrahimrabab

Same issue here. We are using com.azure:azure-storage-blob:12.8.0 and uploading big files using the BlobClient.upload() with InputStreams throws the OutOfMemory error:

Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:239)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at com.azure.storage.common.Utility.lambda$convertStreamToByteBuffer$1(Utility.java:250)
	at com.azure.storage.common.Utility$$Lambda$2025/0x0000000100ecd440.call(Unknown Source)
	at reactor.core.publisher.MonoCallable.call(MonoCallable.java:91)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:401)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:243)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
	at reactor.core.publisher.FluxRange$RangeSubscription.slowPath(FluxRange.java:154)
	at reactor.core.publisher.FluxRange$RangeSubscription.request(FluxRange.java:109)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)
	at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:228)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
	at reactor.core.publisher.FluxRange.subscribe(FluxRange.java:68)
	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
	at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4252)
	at reactor.core.publisher.Mono.block(Mono.java:1684)
	at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:99)
	at com.azure.storage.blob.BlobClient.uploadWithResponse(BlobClient.java:222)
	at com.azure.storage.blob.BlobClient.uploadWithResponse(BlobClient.java:185)
	at com.azure.storage.blob.BlobClient.upload(BlobClient.java:163)

kuikiker avatar Oct 14 '22 12:10 kuikiker

Hi @kuikiker How are you reproducing this error? Can you provide a code snippet, size of files, and heap size of your JVM? Also, did you see this error when you were using an older version of azure-storage-sdk?

ibrahimrabab avatar Oct 14 '22 20:10 ibrahimrabab

Hi @ibrahimrabab

I have just tested it with 1GB heap size (-Xmx1G) and 3 different file sizes.

  • 60MB -> around 300MB heap memory is allocated by the upload method.
  • 120MB -> aournd 400MB is allocated.
  • 300MB -> around 650MB is allocated and the above java.lang.OutOfMemoryError: Java heap space exception is thrown.

This is the upload code snippet. We basically have an InputStream but we send a BufferedStream to the upload.

public static void uploadFile(BlobContainerClient containerClient, String fileName, InputStream inputStream) throws IOException {
    // Get a reference to a blob
    BlobClient blobClient = containerClient.getBlobClient(fileName);
    log.info("Uploading to Blob storage as blob: " + fileName);
    blobClient.upload(new BufferedInputStream(inputStream),inputStream.available(),true);
}

I am now testing newer versions, e.g using com.azure:azure-storage-blob:12.10.0 slightly improves the situation. With 1GB heap memory I can handle 600MB files but it seems that something odd is still there since the heap usage increases alike the files' sizes.

kuikiker avatar Oct 17 '22 06:10 kuikiker

Hi @kuikiker Thank you for providing the information! Some speculations, normally heap size would not grow with larger files, but it is possible it is growing due to larger files performing multiple upload chunks which would cause the heap to grow.

Also, since your max heap size is 1GB, the JVM won't perform garbage collection until necessary. So, it is possible that the heap size growing is due to GC.

If the newer version of blobs helps alleviate the OOM exception, we'd recommend using that over the older versions.

Let us know if this helps or if you have any additional questions! :)

ibrahimrabab avatar Oct 18 '22 22:10 ibrahimrabab

Hi @ibrahimrabab,

We came across similar issue but in staging multiple blocks(chunk size is about 90-100 MB per chunk), the original file could be 1 GB and user split into 10 or 10+ chunks to upload async.

When we try to do some pressure testing about sending 30 chunks requests (it is kind of 3 users upload 1 GB files at the same time), the OOM (java heap space exception) is coming up.

We are using the lib 'com.azure:azure-storage-blob:12.19.0' Related method or code snipped:

async client: BlockBlobAsyncClient blobAsyncClient = buildBlockBlobAsyncClient(); blobAsyncClient.stageBlock(base64BlockId, Flux.just(byteBuffer),bytes.length).block();

Or BlockBlobClient blobClient = buildBlockBlobClient(); ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes); blockBlobClient.stageBlock(base64BlockId, byteArrayInputStream, bytes.length);

JVM setting: -XX:+UseG1GC -Xmx2024m Error: org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1082) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:204) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1787) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3537) at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) at org.springframework.util.StreamUtils.copy(StreamUtils.java:167) at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:112) at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:152) at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest$StandardMultipartFile.getBytes(StandardMultipartHttpServletRequest.java:246) at com.accela.documentservice.service.DocumentServiceImpl.uploadChunk(DocumentServiceImpl.java:46) at com.accela.documentservice.controller.DocumentController.uploadFile(DocumentController.java:67) at com.accela.documentservice.controller.DocumentController$$FastClassBySpringCGLIB$$1e756aad.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:123) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.accela.documentservice.controller.DocumentController$$EnhancerBySpringCGLIB$$c36abd95.uploadFile(<generated>) at java.base/java.lang.invoke.LambdaForm$DMH/0x0000000801400400.invokeVirtual(LambdaForm$DMH) at java.base/java.lang.invoke.LambdaForm$MH/0x0000000801408800.invoke(LambdaForm$MH) at java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:158) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) at java.base/java.lang.reflect.Method.invoke(Method.java:577) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)

Please check if this is a defect or any advice to use this function correctly? Regards Michael

MichaelHoOfficial avatar Nov 04 '22 13:11 MichaelHoOfficial

Some general notes for those on this thread encountering OOM issues:

  1. Handing uploads data in its most versatile form will generally result in better performance. E.g. if you already have the bytes in memory, or from a seekable file, hand them to us in that form (BinaryData.fromBytes(), BinaryData.fromByteBuffer(), BinaryData.fromFile()). The library can more optimally consume this data when it knows its underlying structure.
  2. Providing an InputStream limits what the library can do efficiently (e.g. the library must add large additional buffers when uploading in parallel, as stream reads are sequential).
  3. In general, try to use BinaryData as much as possible to provide your upload contents. The goal is API simplification in not just this library's APIs, but consuming code's APIs as well, without losing the contextual information I mention earlier.
  4. It is highly unlikely we will release memory optimizations as patches for older versions. I recommend updating to latest to determine if the issue is still present. If it is, then discussion can resume on whether this is a usage or a library issue.

jaschrep-msft avatar Nov 29 '22 19:11 jaschrep-msft