redpanda
redpanda copied to clipboard
redpanda-data
Shadow Indexing - Azure Storage support
Who is this for and what problem do they have today?
We want to use shadow indexing but we are an azure cloud customer, as we had discussed with Patrick, we had planned to mitigate that currently only s3 is supported, with using minio gateway, but as the minio project has now removed that functionality, this is not longer tenable. https://github.com/minio/minio/issues/14331
We would as such simply like azure storage support for this feature native.
What are the success criteria?
Able to use azure blob storage for shadow indexing
Why is solving this problem impactful?
We cannot use shadow indexing, as was planning to as such cobbled our data retention. As well was going to use this feature for ability to drive data into lower environments and analytical use cases
Additional notes
@patrickangeles
@michaelandrepearce for our planning, it would be interesting to know how you authenticate with Azure blob storage in your environment. Are you using Managed Identities for VMs, plaintext secrets in Key Vault, something else?
On the AWS side we have IAM roles for this, so looking to work out what the preferred Azure equivalent is.
We run it as private endpoint, so important to support this, e.g. able to change full url as privates use different name than public, and we use access keys which we periodically rotate, so also important to have ability to update this key at runtime, like via admin api,
Azure support with key auth is coming in Redpanda 23.1
In this version, key changes will require a node restart