Retry policy for Azure Blob stores?

[a2f0]

I am using the following dependencies (Django and Azure Blob storage):

django==1.11.15 django-storages==1.7.1 django-storages-azure==1.6.8 azure==4.0.0

It appears whenever I save an image using the .save method on the model's attribute I receive an exception from inside '/site-packages/azure/storage/common/storageclient.py' that says The specified blob does not exist. ErrorCode: BlobNotFound and specifies that Retry policy did not allow for a retry.

The exception also specifies the full URL to the Blob - it's always retrievable if I manually visit the URL. From what I can tell this error is completely benign, and has to do with some sort of write/read consistency issue where metadata is trying to be read back too quickly after the object was written (via a HEAD request against Azure's server). It looks to me that perhaps a retry policy would be the way to fix this, although I do not see any support for retry policies in django-storages. I feel like this is a bug but I am not sure - it definitely clouds up our Sentry logs. Is this perhaps a known issue? I did not find anything in the issue history or on google that suggests other people have this problem.

Reference: https://azure.microsoft.com/en-us/blog/azure-storage-client-library-retry-policy-recommendations/

[altipard]

@deepeeess any solutions found? i have the same issue

[a2f0]

@altipard No, I have not resolved this yet. I've actually revisited it a few times. Honestly I'd be fine with just pruning the sentry log and calling it a day but I've had a difficult time doing that API-side as well.

[a2f0]

I finally got sentry working with this. Here's the config I used to filter the events.

def filter_unwanted_events(event, hint):
    '''
    This is the before_send hook for sentry.  It is explicitly referenced in the sentry_sdk.init call.
    '''
    # https://github.com/jschneier/django-storages/issues/720
    if 'logentry' in event:
        if 'message' in event['logentry']:
            if event['logentry']['message'] == '%s Retry policy did not allow for a retry: %s, HTTP status code=%s, Exception=%s.':
                print("Skipping sending of 'Retry policy' to Azure.")
                return None
    print("Sending event to Sentry.")
    return event

sentry_sdk.init(
    integrations=[DjangoIntegration()],
    dsn=env('DJANGO_SENTRY_DSN'),
    environment=env('AZURE_ENVIRONMENT'),
    before_send=filter_unwanted_events,
    debug=env.bool('DJANGO_SENTRY_DEBUG', default=False),

    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for performance monitoring.
    # We recommend adjusting this value in production,
    traces_sample_rate=1.0,

    # Associate users to errors
    send_default_pii=True
)