pathway
pathway copied to clipboard
Support Azure Blob Storage as a persistence backend
Is your feature request related to a problem? Please describe. Currently, supported persistence backend are local file system and S3. Support for Azure Blob Storage is also required.
Describe the solution you'd like The implementation of Azure persistence backend can be done in the following steps:
- Implement the key-value storage interface for Azure. Since Rust is used as a framework's core language, a Rust crate must be used to provide access to Azure. azure_storage_blobs can be used, as one of the most popular options;
- Support Azure Blob Storage in the DataStorage config and add an option to the persistent storage config.
- Add Azure as a persistence backend option in the Python frontend.
- Finally, add it as an option to the persistence integration tests.
The integration is expected to be non-breaking, as it introduces a new backend via an existing interface.