AzureBlobStorage target container customisation

[Romanx]

The Problem

We have a problem which is that we create a lot of grains, most of which are related.

The default storage provider puts everything into the same container which is usually fine but since our grains are clustered sometimes we need to clean a large amount of grains rapidly which causes us to run into problems if we try to delete by path since we hit limits with azure.

An alternative solution is to create a container per grain group and delete the whole container which means azure can garbage collect them in the background allowing us to not run into that limit.

We're aware this could cause us to run into odd state issues where the grains think they have stored data when they don't but this is a separate issue since to clean up correctly we would have to index/track every grain instance (to know they're related) and then call them all (possibly activating them) to clear up their data which would cause a lot of churn/load especially on azure which we're trying to avoid.

A Solution?

Add a method to AzureBlobStorageOptions which allows you to set a function for building container factories. This gets given the service provider and the resolved options when building the grain storage. The Func is initalized to a builder for the default container factory which acts how the current implementation works.

public class AzureBlobStorageOptions
{
  public Func<IServiceProvider, AzureBlobStorageOptions, IBlobContainerFactory> BuildContainerFactory { get; set; } = static (ServiceProvider provider, AzureBlobStorageOptions options) =>
  {
      return ActivatorUtilities.CreateInstance<DefaultBlobContainerFactory>(provider, options);
  };
}

This will internally store that type to be constructed later, if not provided then we will use a default factory that creates the container during the init phase and reuses the container when requested.

With this approach during configuration we can override the default container factory by doing the following

builder
    .AddAzureBlobGrainStorage(StorageNames.BlobStorage, options =>
    {
        options.BuildContainerFactory = static (ServiceProvider provider, AzureBlobStorageOptions options) =>
        {
            return ActivatorUtilities.CreateInstance<MyCustomContainerFactory>(provider, options);
        };
    })

On construction of the IGrainStorage instance we would use the type (if exists) to construct an instance of the provided ContainerFactory and inject it into the AzureBlobGrainStorage class. https://github.com/dotnet/orleans/blob/19ab11427885fb4d0dde41673f2d3acc14c8385b/src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs#L310-L317

Becomes

    public static class AzureBlobGrainStorageFactory
    {
        public static IGrainStorage Create(IServiceProvider services, string name)
        {
            var optionsMonitor = services.GetRequiredService<IOptionsMonitor<AzureBlobStorageOptions>>();
            var options = optionsMonitor.Get(name);

            IBlobContainerFactory containerFactory = options.BuildContainerFactory(services, options);
            return ActivatorUtilities.CreateInstance<AzureBlobGrainStorage>(services, name, options, containerFactory);
        }
    }

Then lastly changing the IGrainStorage to use the injected container factory to resolve the container before Read/Write/Clear. Since we're not ensuring that we initalized the container there is a chance that it may not exist when we go to Read/Write/Clear. In the case of Read and Clear then currently it just returns done. When writing new data, if the container doesn't exist then it tries to create the container before retrying the write. All of this is current functionality.

The interface for IBlobContainerFactory is:

/// <summary>
/// A factory for building container clients for blob storage
/// </summary>
public interface IBlobContainerFactory
{
    /// <summary>
    /// Build a container for the specific grain type and grain id
    /// </summary>
    /// <param name="grainType">The grain type</param>
    /// <param name="grainId">The grain id</param>
    /// <returns>A configured blob client</returns>
    public BlobContainerClient BuildContainer(string grainType, GrainReference grainId);

    /// <summary>
    /// Initialize any required dependenceis using the provided client and options
    /// </summary>
    /// <param name="client">The connected blob client</param>
    /// <returns>A <see cref="Task"/> representing the asynchronous operation.</returns>
    public Task Init(BlobServiceClient client);
}

I know this was a wall of text but didn't want to just open a PR without a conversation. I'd love to roll this into our own code but unfortunately the overlap with the existing code is so high it'd just be a copy and paste and trying to keep it in sync. Happy to open a PR with the work in both the main and 3.x branches.

[benjaminpetit]

Hi @Romanx , thanks for the suggestion!

Is AzureBlobStorageOptions.ContainerName not sufficient in your case? You would need to create a provider per grain type.

Otherwise, I think a simple Func<string, GrainId> GetContainerNameFunc in AzureBlobStorageOptions would be more simple.

[Romanx]

Hey @benjaminpetit,

We actually would like use dependency injection to access some services hence the approach with using a class. This is across multiple grains and one grain type may have a different container, we encode some information into our grain ids for clustering them and use a custom placement strategy.

The func would work but would perform a capture which i was trying to avoid but may be neglegable.

[Romanx]

I've opened a draft PR with a candidate implementation. As discussed on discord I thought that a specific class implementation rather than a func would allow dependencies to be injected and state to be kept for the container name logic.

In our specific case we have state in our grain ids for clustering so we parse that information out of the grain ids with a specific parser that we want to inject. If the parsing fails (since the grain can't be clustered) then we want to fallback to a default container which we store as state so we don't keep creating the instance of it.

Happy to refactor and chat through the implementation, didn't add any tests yet since I wasn't sure on the direction this would go. Currently only in the main 4.x branch but will backport when settled on the implementation.