[BUG] using OpenWriteAsync with BlockBlobClient does not seems to reuse the BlobClientOptions.Retry options

[miaooss]

Library name and version

Azure.Storage.Blobs 12.13.0

Describe the bug

I'm using BlobClientOptions to add a different retry. It does not seem to be applied to the BlockBlobClient.

public async Task UploadToBlobAsync(string blobSasUrl, Stream sourceStream, CancellationToken cancellationToken)
        {
            BlockBlobClient blob = new(new Uri(blobSasUrl), new BlobClientOptions
            {
                Retry =
                {
                    Mode = RetryMode.Fixed,
                    MaxRetries = 120,
                    Delay = TimeSpan.FromMilliseconds(500),
                    MaxDelay = TimeSpan.FromMilliseconds(5000)
                }
            });
            using Stream blobStream = await blob.OpenWriteAsync(true, null, cancellationToken);

            await sourceStream.CopyToAsync(blobStream, cancellationToken);
        }

Expected behavior

I would expect to have 120 retries in the log and to have 500 ms between them.

Actual behavior

It seems to have only 4 retries with a very short period of time.

Reproduction Steps

I created a URL with Sas token and I'm trying to upload it. (Not the same service as the one creating it)

Environment

Dotnet 6.0, in a container

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[amnguye]

Hm, are those actual retries or are those individual different stage block requests?

Because on 403s I believe the SDK doesn't retry on that error code because that's not a retriable error code. We normally retry on 500s.

[miaooss]

Actually, in our case, the 403 with a share policy sas token can take time to be valid. In this case, the same URL will work later on.

[miaooss]

I'm actually not sure those are different stage block requests or retries

[amnguye]

@JoshLove-msft Was wondering if you could help understand RetryOptions and why it only retried 3 times vs 120 times like the customer set.

I checked the BlockBlobWriteStream which is the Stream that gets returned back from the OpenWriteAync. We just do a simple assignment of saving the client (and the client options involved in the pipeline) in the object. (see https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/src/BlockBlobWriteStream.cs#L42). Let us know if there's anything we needed to change in the storage in order to copy over the retry options correctly.

[JoshLove-msft]

I don't think 403 is a retriable error - https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Common/src/Shared/StorageResponseClassifier.cs#L34

Most likely those are just 3 separate parallel requests.

[miaooss]

Hi,

So as I understand the 403 is not retriable, unfortunately we are using a SAS With policy inside the Blob Storage and the 403 is a common exception we can have. We need some way to retry the call on and not having the call entire call fail.

PS: With this call, I can confirm the stage block is not retried.

[amnguye]

This scenario of retrying on a 403 seems specific to your case. This would be a breaking change to the SDK if we started retrying on 4XX errors and also most 4XX errors should not be retried because they're indicating it was a bad request, and we shouldn't try to send another bad request.

One workaround I can think of is, if 403 is an expected error for your application, I would recommend the workaround of catching the specific Error Code and retrying the request manually. However I know this isn't entirely viable since the Upload API doesn't return back the specific stage block that's failing. But you can also recall the Upload API as well.

I think the better solution from our end, is to implement something that returns the specific stage block(s?) that fail, that way you can retry them. It doesn't look like the BlobDownloadResult gives that information. I will relabel this as a feature request.

I would also recommend waiting for the policy sas token to become valid before attempting to upload, instead of retrying until the policy becomes valid. Just to clarifiy, it's a Stored Access Policy ?

[miaooss]

We are using a StoredAccessPolicy: https://learn.microsoft.com/en-us/azure/storage/common/storage-stored-access-policy-define-dotnet?tabs=dotnet#create-a-stored-access-policy

We are waiting for the ((BlockBlobClient)blob).OpenWriteAsync(true) to not return an exception (In our case the 403). Once we have a successful write we proceed with the Upload.

You mention that this is not an effective way to validate the sas. What would you recommend as a better way?

Thank you.

[miaooss]

Is there any way to just override the stream return to retry on this specific error?

[christianh25]

amnguye

I would also recommend waiting for the policy sas token to become valid before attempting to upload, instead of retrying until the policy becomes valid. Just to clarifiy, it's a Stored Access Policy ?

How do you detect that the policy is now valid without blindly waiting 30 seconds?

[navba-MSFT]

Adding Service team to look into this.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details

---

Library name and version

Azure.Storage.Blobs 12.13.0

Describe the bug

I'm using BlobClientOptions to add a different retry. It does not seem to be applied to the BlockBlobClient.

public async Task UploadToBlobAsync(string blobSasUrl, Stream sourceStream, CancellationToken cancellationToken)
        {
            BlockBlobClient blob = new(new Uri(blobSasUrl), new BlobClientOptions
            {
                Retry =
                {
                    Mode = RetryMode.Fixed,
                    MaxRetries = 120,
                    Delay = TimeSpan.FromMilliseconds(500),
                    MaxDelay = TimeSpan.FromMilliseconds(5000)
                }
            });
            using Stream blobStream = await blob.OpenWriteAsync(true, null, cancellationToken);

            await sourceStream.CopyToAsync(blobStream, cancellationToken);
        }

Expected behavior

I would expect to have 120 retries in the log and to have 500 ms between them.

Actual behavior

It seems to have only 4 retries with a very short period of time.

Reproduction Steps

I created a URL with Sas token and I'm trying to upload it. (Not the same service as the one creating it)

Environment

Dotnet 6.0, in a container

Author:	miaooss
Assignees:	-
Labels:	Storage, Service Attention, Client, customer-reported, feature-request, needs-team-attention
Milestone:	-