Azure Blob Storage configuration does not honor azure.endpoint-suffix configuration

[Pete-PlaytimeSolutions]

trafficstars

Describe the bug Configuring Tempo to use an azure blob storage in a region other than Azure global(blob.core.windows.net) is not working. The Azure configuration should support other regions, such as

• Azure China(blob.core.chinacloudapi.cn),
• Azure German(blob.core.cloudapi.de),
• Azure US Government(blob.core.usgovcloudapi.net).
• local Azure Blob Storage emulator

To Reproduce Steps to reproduce the behavior:

1. configured Tempo using an Azure Blob Storage account with azure.endpoint_suffix=blob.core.usgovcloudapi.net
2. Tried staring Tempo
3. Tempo crashes, with error failed to authenticate request to "https://XXXX.blob.core.windows.net", as it is trying to hit a service in the wrong region...

Expected behavior When Tempo is configured using the endpoint_suffix option storage_config: azure: endpoint_suffix: blob.core.usgovcloudapi.net

Then all Blob Storage API calls should use this configuration.

see: Tempo Storage Documentation azure endpoint-suffix

Environment:

• any

[joe-elliott]

Thanks for reporting this issue.

Can you share your complete azure config block? Also any relevant log messages would help.

[Pete-PlaytimeSolutions]

storage:
  trace:
    backend: azure
    blocklist_poll_tenant_index_builders: 1
    blocklist_poll_jitter_ms: 500
    azure:
      storage-account-name: devstoreaccount1
      storage-account-key: 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=='
      endpoint-suffix: localhost:10000
      container-name: traces
      # use-managed-identity: true
      # hedge-requests-at: 500ms
      # hedge-requests-up-to: 2

NOTE: This works fine when I configure an Acct name and key, located in the blob.core.windows.net region. And the same config setup works fine for both Loki and Mimir containers...There was a similar issue in Loki, but that has recently been fixed. see #6842

I don't have access to the exact error message at the moment, but basically it states it can't authenticate to site devstoreaccount1.blob.core.windows.net, as the account isn't registered for that region.

[joe-elliott]

is that storage-account-key sensitive? if so, may want to remove.

Thanks for the details and the link to the Loki fix. That should definitely help.

Looks like we're using that config value here:

https://github.com/grafana/tempo/blob/main/tempodb/backend/azure/azure_helpers.go#L91 https://github.com/grafana/tempo/blob/main/tempodb/backend/azure/azure_helpers.go#L186

Hopefully the fix will be fairly easy. I'll tag this up to have someone take a look when we can.

[Pete-PlaytimeSolutions]

The key is fine, its the standard default one used for the azure blob storage emulator

[electron0zero]

I added a test in https://github.com/grafana/tempo/pull/1773 to reproduce the issue and I was not able to reproduce it in the unit tests. It seems to be building URLs correctly as per Azure docs.

@Pete-PlaytimeSolutions can you share logs or a setup to reproduce the issue so I can debug it further?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.