Azure - Authentication error when `AZURE_CUSTOM_DOMAIN` set to Azure CDN

[JeffreyCA]

Ever since the Azure backend was updated to the new azure-storage-blob library in v1.12, using django-storages with AZURE_CUSTOM_DOMAIN set results in Authentication errors when uploading files. For me this only happens with Akamai CDNs, but as reported by others below it affects other CDN types as well.

v1.12 changed how AZURE_CUSTOM_DOMAIN is used with BlobServiceClient. In v1.11 and earlier, the custom domain was only used to get blob URLs. All other operations like uploading, streaming, getting metadata were being done by making requests to the actual storage account endpoint (https://<accountname>.blob.core.windows.net) even if a custom domain was specified.

In v1.12, the behaviour changed so that the custom domain endpoint is used for all storage operations. This uncovered several different upstream issues causing various storage requests to fail with auth errors.

Issues

1. Auth error due to MAC signature mismatch when AZURE_CUSTOM_DOMAIN set to Akamai CDN

Upstream issue: https://github.com/Azure/azure-sdk-for-python/issues/26381

Uploads fail with the following error:

ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
...
authenticationerrordetail:The MAC signature found in the HTTP request '5dC3N7RcRW9V...' is not the same as any computed signature. Server used following string to sign: 'PUT


1

application/octet-stream






x-ms-blob-type:BlockBlob
x-ms-client-request-id:xxxx
x-ms-date:Mon, 21 Feb 2022 07:34:35 GMT
x-ms-version:2020-10-02
Content: <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed 
correctly including the signature.
RequestId:xxx
Time:2022-02-21T07:34:35.1058206Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request '5dC3N7RcRW9V...' is not the same as any computed 
signature. Server used following string to sign: 'PUT

2. Forbidden ClientAuthenticationError when AZURE_CUSTOM_DOMAIN set to Microsoft CDN

Upstream issue: https://github.com/Azure/azure-sdk-for-python/issues/23640

Uploads fail with the following error (different than above):

azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden'
ErrorCode:AuthenticationFailed

[JeffreyCA]

Opened an issue on Azure Python SDK repo here: https://github.com/Azure/azure-sdk-for-python/issues/23163

[madalinpopa]

I have the same issue. Once I update the AZURE_CUSTOM_DOMAIN with my *.azureedge.net domain, I get the error below when run python manage collectstatic.

Traceback (most recent call last):
  File "/home/support/repos/pythonazurestocdn/mysite/manage.py", line 22, in <module>
    main()
  File "/home/support/repos/pythonazurestocdn/mysite/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 425, in execute_from_command_line
    utility.execute()
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 373, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 187, in handle
    collected = self.collect()
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
    handler(path, prefixed_path, storage)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 334, in copy_file
    if not self.delete_file(path, prefixed_path, source_storage):
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 248, in delete_file
    if self.storage.exists(prefixed_path):
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/storages/backends/azure_storage.py", line 241, in exists
    blob_client.get_blob_properties()
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/azure/storage/blob/_blob_client.py", line 1242, in get_blob_properties
    process_storage_error(error)
  File "/home/support/repos/pythonazurestocdn/.venv/lib/python3.9/site-packages/azure/storage/blob/_shared/response_handlers.py", line 177, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
  File "<string>", line 1, in <module>
azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden'
ErrorCode:AuthenticationFailed

[jschneier]

The upstream issue response seems to indicate this is no longer a problem?

[justinpitcher]

It's still happening. We are experiencing the same issue with CDNs and django-storages 1.12.3

[dimbleby]

if there's a standalone repro that doesn't involve django-storages, then this is the wrong place to track the problem. Suggest taking it back upstream.

[tarak]

The BlobServiceClient uses the custom domain URL, if AZURE_CUSTOM_DOMAIN is set. This obviously won't work for uploading files. See:

https://github.com/jschneier/django-storages/blob/5015c243473372d1904761cd6d8ef8fdb384df31/storages/backends/azure_storage.py#L153

IMHO, saving files should use the storage account URL (ending in *.blob.core.windows.net), while the custom domain should only be used if set and return the URL from that custom domain.

Simply put, one cannot write to an Azure CDN endpoint. I may be wrong though...

[tarak]

I think that the documentation is misleading. I use an Azure CDN that uses a storage account as endpoint. After I have read the documentation of the Azure backend I thought I could set the AZURE_CUSTOM_DOMAIN setting to the hostname of the CDN endpoint (e.g. edgeaccount.azureedge.net). But now I think that the AZURE_CUSTOM_DOMAIN actually means the custom domain of the storage account itself. If so, please ignore my previous comment...

[jeffreykirchner]

I agree with @tarak, docs say AZURE_CUSTOM_DOMAIN can be mycdn.azureedge.net.

It would also make sense for the AZURE_CONNECTION_STRING setting to not override the AZURE_CUSTOM_DOMAIN. Files should be uploaded with the AZURE_CONNECTION_STRING settings and return the AZURE_CUSTOM_DOMAIN url.

Another option would be to add an AZURE_CDN setting that accomplishes the above. Most people are using Azure CDN and Azure Storage in concert.

[milopersic]

Having the same issue as described here, setting AZURE_CUSTOM_DOMAIN to the CDN endpoint breaks uploads for my application, but setting it to the storage container itself serves files from the container not the CDN.

[JeffreyCA]

Is everyone who has this issue using Akamai CDNs? Regarding the auth error I was having, I discovered the root issue to be with how Akamai CDN handles the If-None-Match HTTP header, which the new Azure SDKs are sending: https://github.com/Azure/azure-sdk-for-python/issues/26381. Not sure if this will be fixed on the SDK side or CDN side.

While I think Azure SDKs are supposed to work with custom domains and CDNs (see https://github.com/Azure/azure-sdk-for-python/issues/25536), there are multiple issues affecting different Azure CDN types which make using CDN endpoints quite unreliable at the moment. django-storages v1.11 (old Azure SDK) uses the storage account URL for API calls, bypassing the custom domain. I think it makes sense to restore that behaviour and have the custom domain only be used for getting the blob URL. This is how the old Azure and S3 backends work.

Issues identified when connecting using CDN endpoint instead of storage URL

Microsoft CDN

• Streaming large files leads to IncompleteReadError: https://github.com/jschneier/django-storages/issues/1097
  • Issue closed but provided workaround does not work.
• ClientAuthenticationError when calling exists(): https://github.com/Azure/azure-sdk-for-python/issues/23640
  • Issue closed with workaround being "use a different CDN type".

Akamai CDN

• AuthenticationFailed when uploading: https://github.com/Azure/azure-sdk-for-python/issues/26381

[jeffreykirchner]

I am using Microsoft CDN + Azure Storage Account. I agree, with your proposed fix, see my comment above.

[dimbleby]

Did you report the problem that opens this thread to the Azure SDK repository? Since that repro didn't involve django-storages at all, it's clear that any fix would need to be upstream.

Of course it might make sense to implement some workaround while waiting for upstream to sort this. But this is not the only codebase that uses blob storage - it's better for the world if this can be fixed at root!

[jeffreykirchner]

The issue is that when STATICFILES_STORAGE is defined, it is using AZURE_CUSTOM_DOMAIN to connect. DEFAULT_FILE_STORAGE is working correctly and does not use AZURE_CUSTOM_DOMAIN to connect. Perhaps it would make sense to add a new AZURE_CDN_URL parameter that would handle CDN endpoints. That would leave AZURE_CUSTOM_DOMAIN to define an actual custom domain (and not an endpoint URL).

[JeffreyCA]

I created a PR #1176 to revert back to the old behaviour so that AZURE_CUSTOM_DOMAIN is used only for generating file URLs. This is in line with the other storage backends. I also updated the docs to clarify this

[mlindemu]

@jschneier , when will we get a new PyPi release with this change rolled in? Thank you for the great library!

[mlindemu]

For anyone following or looking at this thread, the fix is in the published 1.13.2 version. Thank you!