[BUG] StacklessClosedChannelException on Blob Downloads in Windows Azure Functions on Version 12.25.4

[SoulKa]

Describe the bug I am getting io.netty.channel.StacklessClosedChannelException exceptions when downloading blobs using downloadToFile() in version 12.25.4. The same code works using version 12.25.3. It is running in an Azure Function. The download link is valid and the blob storage is available. I can reproduce the error. Probably an issue due to the netty upgrade in version 12.25.4?

Exception or Stack Trace

Exception: io.netty.channel.StacklessClosedChannelException
Stack: reactor.core.Exceptions$ReactiveException: io.netty.channel.StacklessClosedChannelException
	at reactor.core.Exceptions.propagate(Exceptions.java:396)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:98)
	at reactor.core.publisher.Mono.block(Mono.java:1742)
	at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:133)
	at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1208)
	at com.azure.storage.blob.specialized.BlobClientBase.downloadToFileWithResponse(BlobClientBase.java:1169)
	at com.azure.storage.blob.specialized.BlobClientBase.downloadToFile(BlobClientBase.java:1078)
	< CENSORED USER CLASSES >
	at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
	at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:22)
	at com.microsoft.azure.functions.worker.chain.FunctionExecutionMiddleware.invoke(FunctionExecutionMiddleware.java:19)
	at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
	at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invoke(JavaFunctionBroker.java:133)
	at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:124)
	at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:34)
	at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
	at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:44)
	at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:94)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)
		... 28 more
Caused by: io.netty.channel.StacklessClosedChannelException
	at io.netty.channel.AbstractChannel$AbstractUnsafe.ensureOpen(ChannelPromise)(Unknown Source)
	Suppressed: io.netty.channel.StacklessClosedChannelException
		... 1 more
	Suppressed: io.netty.channel.StacklessClosedChannelException
		... 1 more
	Suppressed: io.netty.channel.StacklessClosedChannelException
		... 1 more

To Reproduce Steps to reproduce the behavior:

1. Create an Azure Function that creates a blob client using new BlobClientBuilder().endpoint(blobSasUrl).buildClient() and a valid download URL to a blob with a valid SAS URL. Version 12.25.4 of the storage library has to be used.
2. Download an existing blob with client.downloadToFile(destinationPath, true)
3. Get the StacklessClosedChannelException

Code Snippet

var client = new BlobClientBuilder().endpoint(blobSasUrl).buildClient();
client.downloadToFile(destinationPath, true);

Expected behavior The blob gets successfully downloaded as it was in version 12.25.3

Setup (please complete the following information):

• OS: Windows
• IDE: IntelliJ Ultimate
• Library/Libraries: com.azure:azure-sdk-bom:1.2.23
  • com.azure:azure-core-http-netty:1.14.2
  • com.azure:azure-storage-blob:12.25.4
• Java version: 17
• App Server/Environment: Azure Function running on Runtime 4.33.1.22394 and App Service Plan EP2 and EP3, both getting the same issue
• Frameworks: None, only Azure Functions SDK

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [X] Bug Description Added
• [X] Repro Steps Added
• [X] Setup information Added

[github-actions[bot]]

@ibrahimrabab @ibrandes @seanmcc-msft

[github-actions[bot]]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[SoulKa]

I just ran the function locally on my mac. I could not reproduce the error there. It has to do with the Windows Function environment in combination with the new library...

[SoulKa]

Update: I noticed that the HTTP download of the blob simply does not progress. So the blob storage closes the TCP/HTTP connection after ~45s thus the exception. It seems that this is the origin of the issue. I do not know the internals of the Azure netty or blob storage download implementation, but some change in 12.25.4 results in the freezing HTTP call.

Please tell me if I can provide any useful information or tests.

[SoulKa]

Is someone working on this? Azure Functions seem like a common use case for blob downloads. It would be bad if this issue persists in the newer versions.

[ibrahimrabab]

Hi @SoulKa We are investigating this issue on our end and will get back to you on this thread as soon as we can with our findings. Thank you for your patience!

[SoulKa]

Thank you for investigating this issue! I can provide further details where needed

[ibrahimrabab]

Hi @SoulKa

Wanted to confirm, have you used either of these guides when trying to develop with Storage and Azure Functions?

https://learn.microsoft.com/en-us/azure/developer/java/sdk/troubleshooting-dependency-version-conflict#configure-azure-functions

https://learn.microsoft.com/en-us/azure/developer/java/sdk/troubleshooting-dependency-version-conflict#create-a-fat-jar

[SoulKa]

Hi @ibrahimrabab,

I have not used these guides yet. Are they relevant when I do not have a netty dependency in my project? These are my dependencies:

pom.xml

  <dependencies>
    <dependency>
      <groupId>com.microsoft.azure.functions</groupId>
      <artifactId>azure-functions-java-library</artifactId>
      <version>3.1.0</version>
    </dependency>

    <dependency>
      <groupId>com.azure</groupId>
      <artifactId>azure-storage-blob</artifactId>
    </dependency>

    <!-- Test -->
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter</artifactId>
      <version>5.10.2</version>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.mockito</groupId>
      <artifactId>mockito-core</artifactId>
      <version>5.11.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.mockito</groupId>
      <artifactId>mockito-junit-jupiter</artifactId>
      <version>5.11.0</version>
      <scope>test</scope>
    </dependency>

    <!-- Lombok as QoL Dependency -->
    <dependency>
      <artifactId>lombok</artifactId>
      <groupId>org.projectlombok</groupId>
      <scope>provided</scope>
      <version>1.18.32</version>
    </dependency>

    <dependency>
      <artifactId>jackson-databind</artifactId>
      <groupId>com.fasterxml.jackson.core</groupId>
      <version>2.17.1</version>
    </dependency>

    <dependency>
      <artifactId>jackson-datatype-jsr310</artifactId>
      <groupId>com.fasterxml.jackson.datatype</groupId>
      <version>2.17.1</version>
    </dependency>

    <dependency>
      <artifactId>annotations</artifactId>
      <groupId>org.jetbrains</groupId>
      <scope>provided</scope>
      <version>24.1.0</version>
    </dependency>
  </dependencies>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-sdk-bom</artifactId>
        <version>1.2.22</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

[ibrahimrabab]

@SoulKa While your POM doesn't explicitly include Netty, since it's a dependency of azure-storage-blob it will be included in the application. So, the guides shared are relevant.

Once you go through the guides, please let us know if you are still facing the issue, thank you!

[iron-condor]

Bumping this issue. I am having the same trouble with azure-storage-blob version 12.29.0 while trying to write to a blob. After reverting to version 12.25.4, the issue no longer manifests, and writes are successful.

[kristoforlovsky]

Hello simiralry I got the same problem and the same outcome. For me to resolve the issue the steps were: I started of with newest Azure Java SDK BOM file at the time 1.2.31 and 3 dependencies - azure-storage-blob, azure-storage-common and azure-identity. I used this setup due to Microsoft Learn page recommendations. The BOM used version 12.29.0.. What worked for me was to remove the BOM and use direct dependency.. I took inspiration from thi post and straightly used 12.25.3 So only one direct dependency azure-storage-blob on version 12.25.3. This immediately solved my problem. Also the OS i am testing it on is Linux based so not just Windows problem I guess.. I hope this helps someone - I've been trying solutions for several days.. and honestly this maybe wont be it for me because on this version I got dependency conflict with older already in use library...