Epic: IDistributedCache updates in .NET 9

[adityamandaleeka]

Update:

HybridCache has relocated to dotnet/extensions:dev; it does not ship in .NET 9 RC1, as a few missing and necessary features are still in development; however, we expect to ship either alongside or very-shortly-after .NET 9! ("extensions" has a different release train, that allows additional changes beyond the limit usually reserved for in-box packages; HybridCache has always been described as out-of-box - i.e. a NuGet package - so: there is no reason for us to limit ourselves by the runtime restrictions)

---

Status: feedback eagerly sought

• API proposal part 1, core API: https://github.com/dotnet/aspnetcore/issues/54647 and https://github.com/dotnet/aspnetcore/pull/55084
• Initial cut of proposal part 1: https://github.com/dotnet/aspnetcore/pull/55147 and fixes: https://github.com/dotnet/aspnetcore/pull/55251
• API proposal part 2, tags and invalidation: https://github.com/dotnet/aspnetcore/issues/55308
• By nomenclature: https://github.com/dotnet/aspnetcore/issues/55332

Tl;Dr

• add a new HybridCache API (and supporting pieces) to support more convenient and efficient distributed cache usage
• support read-through caching with lambda callbacks
• support flexible serialization
• support stampede protection
• support L1/L2 cache scenarios
• build on top of IDistributedCache so that all existing cache backends work without change (although they could optionally add support for new features)
• support comparable expiration concepts to IDistributedCache

Problem statement

The distributed cache in asp.net (i.e. IDistributedCache) is not particularly developed; it is inconvenient to use, lacks many desirable features, and is inefficient. We would like this API to be a "no-brainer", easy to get right feature, making it desirable to use - giving better performance, and a better experience with the framework.

Typical usage is shown here; being explicit about the problems:

Inconvenient usage

The usage right now is extremely manual; you need to:

• attempt to read a stored value (as byte[])
• check that value for null ("no value")
  • if not null:
    • fetch the value
    • serialize it
    • store the value
  • return the value

This is a lot of verbose boilerplate, and while it can be abstracted inside projects using utility methods (often extension methods), the vanilla experience is very poor.

Inefficiencies

The existing API is solely based on byte[]; the demand for right-sized arrays means no pooled buffers can be used. This broadly works for in-process memory-based caches, since the same byte[] can be returned repeatedly (although this implicitly assumes the code doesn't mutate the data in the byte[]), but for out-of-process caches this is extremely inefficient, requiring constant allocation.

Missing features

The existing API is extremely limited; the concrete and implementation-specific IDistributedCache implementation is handed directly to callers, which means there is no shared code reuse to help provide these features in a central way. In particular, there is no mechanism for helping with "stampede" scenarios - i.e. multiple concurrent requests for the same non-cached value, causing concurrent backend load for the same data, whether due to a cold-start empty cache, or key invalidation. There are multiple best-practice approaches that can mitigate this scenario, which we do not currently employ.

Likewise, we currently assume an in-process or out-of-process cache implementation, but caching almost always benefits from multi-tier storage, with a limited in-process (L1) cache supplemented by a separate (usually larger) out-of-process (L2) cache; this gives the "best of both" world, where the majority of fetches are served efficiently from L1, but cold-start and less-frequently-accessed data still doesn't hammer the underlying backend, thanks to L2. Multi-tier caching can sometimes additionally exploit cache-invalidation support from the L2 implementation, to provide prompt L1 invalidation as required.

This epic proposes changes to fill these gaps

Current code layout

At the moment the code is split over multiple components, in the main runtime, asp.net, and external packages (only key APIs shown):

• Microsoft.Extensions.Caching.Abstractions

  • src: https://github.com/dotnet/runtime/tree/main/src/libraries/Microsoft.Extensions.Caching.Abstractions
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Abstractions/
  • adds IDistributedCache
  • adds DistributedCacheEntryOptions

• Microsoft.Extensions.Caching.Memory

  • src: https://github.com/dotnet/runtime/tree/main/src/libraries/Microsoft.Extensions.Caching.Memory
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Memory/
  • adds AddDistributedMemoryCache
  • adds MemoryDistributedCache : IDistributedCache

• Microsoft.Extensions.Caching.StackExchangeRedis

  • src: https://github.com/dotnet/aspnetcore/tree/main/src/Caching/StackExchangeRedis
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.StackExchangeRedis
  • adds AddStackExchangeRedisCache

• Microsoft.Extensions.Caching.SqlServer

  • src: https://github.com/dotnet/aspnetcore/tree/main/src/Caching/SqlServer
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.SqlServer
  • adds AddDistributedSqlServerCache

• Microsoft.Extensions.Caching.Cosmos

  • src: https://github.com/Azure/Microsoft.Extensions.Caching.Cosmos
  • pkg: https://www.nuget.org/packages/Microsoft.Extensions.Caching.Cosmos
  • adds AddCosmosCache

• Alachisoft.NCache.OpenSource.SDK

  • src: https://www.alachisoft.com/ncache/
  • pkg: https://www.nuget.org/packages/Alachisoft.NCache.OpenSource.SDK/
  • adds AddNCacheDistributedCache

• AWS

  • https://github.com/awslabs/aws-dotnet-distributed-cache-provider

This list is not exhaustive - other 3rd-party and private implementations of IDistributedCache exist, and we should avoid breaking the world.

Proposal

The key proposal here is to add a new caching abstraction that is more focused, HybridCache, in Microsoft.Extensions.Caching.Abstractions; this API is designed to act more as a read-through cache, building on top[ of the existing IDistributedCache implementation, providing all the implementation details required for a rich experience. Additionally, while simple defaults are provided for the serializer, it is an explicit aim to make such concerns fully configurable, allowing for json, protobuf, xml, etc serialization as appropriate to the consumer.

namespace Microsoft.Extensions.Caching.Distributed;

public abstract class HybridCache // default concrete impl provided by service registration
{
    protected HybridCache() { }

    // read-thru usage
    public abstract ValueTask<T> GetOrCreateAsync<TState, T>(string key, TState state, Func<TState, CancellationToken, ValueTask<T>> callback, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);
    public virtual ValueTask<T> GetOrCreateAsync<T>(string key, Func<CancellationToken, ValueTask<T>> callback,
    HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default)
    { /* shared default implementation uses TState/T impl */ }

    // manual usage
    public abstract ValueTask<(bool Exists, T Value)> GetAsync<T>(string key, HybridCacheEntryOptions? options = null, CancellationToken cancellationToken = default);
    public abstract ValueTask SetAsync<T>(string key, T value, HybridCacheEntryOptions? options = null, ReadOnlyMemory<string> tags = default, CancellationToken cancellationToken = default);

    // key invalidation
    public abstract ValueTask RemoveKeyAsync(string key, CancellationToken cancellationToken = default);
    public virtual ValueTask RemoveKeysAsync(ReadOnlyMemory<string> keys, CancellationToken cancellationToken = default)
    { /* shared default implementation uses RemoveKeyAsync */ }

    // tag invalidation
    public virtual ValueTask RemoveTagAsync(string tag, CancellationToken cancellationToken = default)
    { /* shared default implementation uses RemoveTagsAsync */ }
    public virtual ValueTask RemoveTagsAsync(ReadOnlyMemory<string> tags, CancellationToken cancellationToken = default) => default;
}

Notes:

• the intent is that instead of requesting IDistributedCache, consumers might use HybridCache; to enable this, the consumer must additionally perform a services.AddHybridCache(...); step during registration
• the naming of GetOrCreateAsync<T> is for parity with MemoryCache.GetOrCreateAsync<T>
• RemoveAsync and RefreshAsync mirror the similarIDistributedCache methods
• it is expected that the callback (when invoked) will return a non-null value; consistent with MemoryCache et-al, null is not a supported value, and an appropriate runtime error will be raised

Usage of this API is then via a read-through approach using lambda; the simplest (but slightly less efficient) approach would be simply:

// HybridCache injected via DI
var data = await cache.GetOrCreateAsync(key, _ => /* some backend read */, [expiration etc], [cancellation]);

In this simple usage, it is anticipated that "captured variables" etc are used to convey the additional state required, as is common for lambda scenarios. A second "stateful" API is provided for more advanced scenarios where the caller wishes to trade convenience for efficiency; this usage is slightly more verbose but will be immediately familiar to the users who would want this feature:

// HybridCache injected via DI
var data = await cache.GetOrCreateAsync(key, (some state here), static (state, _) => /* some backend read */, [expiration etc], [cancellation]);

This has been prototyped and works successfully with type inference etc.

The implementation (see later) deals with all the backend fetch, testing, serialization etc aspects internally.

(in both examples, the "discard" (_) is conveying the CancellationToken for the backend read, and can be used by providing a receiving lambda parameter)

An internal implementation of this API would be registered and injected via a new AddHybridCache API (Microsoft.Extensions.Caching.Abstractions):

namespace Microsoft.Extensions.Caching.Distributed;

public static class HybridCacheServiceExtensions
{
    public static IServiceCollection AddHybridCache(this IServiceCollection services, Action<HybridCacheOptions> setupAction)
    {...}

    public static IServiceCollection AddHybridCache(this IServiceCollection services)
    {...}
}

The internal implementation behind this would receive IDistributedCache for the backend, as it exists currently; this means that the new implementation can use all existing distributed cache backends. By default, AddDistributedMemoryCache is also assumed and applied automatically, but it is intended that this API be effective with arbitrary IDistributedCache backends such as redis, SQL Server, etc. However, to address the issue of byte[] inefficiency, a new entirely optional API is provided and tested for; if the new backend is detected, lower-allocation usage is possible. This follows the pattern used for output-cache in net8:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IBufferDistributedCache : IDistributedCache
{
    ValueTask<CacheGetResult> GetAsync(string key, IBufferWriter<byte> destination, CancellationToken cancellationToken);
    ValueTask SetAsync(string key, ReadOnlySequence<byte> value, DistributedCacheEntryOptions options, CancellationToken cancellationToken);
}

public readonly struct CacheGetResult
{
    public CacheGetResult(bool exists);
    public CacheGetResult(DateTime expiry);

    public CacheGetResult(TimeSpan expiry);

    public bool Exists { get; }
    public TimeSpan? ExpiryRelative { get; }
    public DateTime? ExpiryAbsolute { get; }
}

(the intent of the usual members here is to convey expiration in the most appropriate way for the backend, relative vs absolute, although only one can be specified; the internals are an implementation detail, likely to use overlapped 8-bytes for the DateTime/TimeSpan, with a discriminator)

In the event that the backend cache implementation does not yet implement this API, the byte[] API is used instead, which is exactly the status-quo, so: no harm. The purpose of CacheGetResult is to allow the backend to convey backend expiration information, relevant for L1+L2 scenarios (design note: async precludes out TimeSpan?; tuple-type result would be simpler, but is hard to tweak later). The expiry is entirely optional and some backends may not be able to convey it, and we need to handle it lacking when IBufferDistributedCache is not supported - in either event, the inbound expiration relative to now will be assumed for L1 - not ideal, but the best we have.

Serialization

For serialization, a new API is proposed, designed to be trivially implemented by most serializers - again, preferring modern buffer APIs:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializer<T>
{
    T Deserialize(ReadOnlySequence<byte> source);
    void Serialize(T value, IBufferWriter<byte> target);
}

Inbuilt handlers would be provided for string and byte[] (and possibly BinaryData if references allow); an extensible serialization configuration API supports other types - by default, an inbuilt object serializer using System.Text.Json would be assumed, but it is intended that alternative serializers can be provided globally or per-type. This is likely to be for more efficient bandwidth scenarios, such as protobuf (Google.Protobuf or protobuf-net) etc, but could also be to help match pre-existing serialization choices. While manually registering a specific IHybridCacheSerializer<Foo> should work, it is also intended to generalize the problem of serializer selection, via an ordered set of serializer factories, specifically by registering some number of:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IHybridCacheSerializerFactory
{
    bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer);
}

By default, we will register a specific serializer for string, and a single factory that uses System.Text.Json, however external library implementations are possible, for example:

namespace Microsoft.Extensions.Caching.Distributed;

[SuppressMessage("ApiDesign", "RS0016:Add public types and members to the declared API", Justification = "demo code only")]
public static class ProtobufDistributedCacheServiceExtensions
{
    public static IServiceCollection AddHybridCacheSerializerProtobufNet(this IServiceCollection services)
    {
        ArgumentNullException.ThrowIfNull(services);
        services.AddSingleton<IHybridCacheSerializerFactory, ProtobufNetSerializerFactory>();
        return services;
    }

    private sealed class ProtobufNetSerializerFactory : IHybridCacheSerializerFactory
    {
        public bool TryCreateSerializer<T>([NotNullWhen(true)] out IHybridCacheSerializer<T>? serializer)
        {
            // in real implementation, would use library rules
            if (Attribute.IsDefined(typeof(T), typeof(DataContractAttribute)))
            {
                serializer = new ProtobufNetSerializer<T>();
                return true;
            }
            serializer = null;
            return false;
        }
    }
    internal sealed class ProtobufNetSerializer<T> : IHybridCacheSerializer<T>
    {
        // in real implementation, would use library serializer
        public T Deserialize(ReadOnlySequence<byte> source) => throw new NotImplementedException();

        public void Serialize(T value, IBufferWriter<byte> target) => throw new NotImplementedException();
    }
}

The internal implementation of HybridCache would lookup T as needed, caching locally to prevent constantly using the factory API.

Additional functionality

The internal implementation of HybridCache should also:

• hold the necessary state to serve concurrent requests for the same key from the same incomplete task, similar to the output-cache implementation
• hold the necessary state to support L1/L2 caching
• optionally, support L1 invalidation by a new optional invalidation API

Note that it is this additional state for stampede and L1/L2 scenarios (and the serializer choice, etc) that makes it impractical to provide this feature simply as extension methods on the existing IDistributedCache.

The new invalidation API is anticipated to be something like:

namespace Microsoft.Extensions.Caching.Distributed;

public interface IDistributedCacheInvalidation : IDistributedCache
{
    event Func<string, ValueTask> CacheKeyInvalidated;
}

(the exact shape of this API is still under discussion)

When this is detected, the event would be subscribed to perform L1 cache invalidation from the backend.

Additional things to be explored for HybridCacheOptions:

• options for L1 / L2 caching; perhaps enabled by default if we have IDistributedCacheInvalidation ?
• eager pre-fetch, i.e. "you've asked for X, and the L1 value is still valid, but only just; I'll give you the L1 value, but I'll kick off a fetch against the backend, so there is not a delay when it expires shortly" (disabled by default, due to concerns over lambdas and captured state mutation)
• compression (disabled by default, for simple compatibility with existing data)
• ...?

Additional modules to be enhanced

To validate the feature set, and to provide the richest experience:

• Microsoft.Extensions.Caching.StackExchangeRedis should gain support for IBufferDistributedCache and IDistributedCacheInvalidation - the latter using the "server-assisted client-side caching" feature in Redis
• Microsoft.Extensions.Caching.SqlServer should gain support for IBufferDistributedCache, if this can be gainful re allocatiuons
• guidance should be offered to the Microsoft.Extensions.Caching.Cosmos owners, and if possible: Alachisoft.NCache.OpenSource.SDK

Open issues

• ~~does the approach sound agreeable?~~
• ~~naming~~
• where (in terms of packages) does the shared implementation go? in particular, it may need access to System.Text.Json, and possible an L1 implementation ( which could be System.Runtime.Caching, Microsoft.Extensions.Caching.Memory, this new one, or something else) and possibly compression; maybe a new Microsoft.Extensions.Caching.Distributed ? but if so, should it be in-box with .net, or just NuGet? or somewhere else?
• the exact choice of L1 cache (note: this should be an implementation detail; we don't need L1+L2 for MVP)
• ~~how exactly to configure the serializer~~
• ~~options for eager pre-fetch TTL and enable/disable L1+L2, via TypedDistributedCacheOptions~~
• ~~should we add tagging support at this juncture?~~

[aKzenT]

I know it's difficult to change the existing interface, but depending on the other changes planned in this epic, it would be great if we could have distributed cachr methods that were more performance oriented, working with Span instead of byte[].

A common scenario for using the cache is saving serialized objects or large texts encoded as UTF-8. Requiring byte[] usually means copying this data at least once.

Similar issues exist when reading from the cache which also returns a byte[] and thus does not allow for using rented buffers or similar optimizations.

As the cache is often used several times for each request in any non trivial web application (e.g. session store, query cache, response cache), optimizations here would really pay off.

[mgravell]

Yup. Totally understand, @aKzenT , and that's part of the prototype. Will update with more details of the proposed API as it evolves, but the short version here is:

• serialization will be moved inside the cache layer
• pluggable serialization to be based on ReadOnlySequence-byte and IBufferWriter-byte
• new optional backend (store) API to allow buffer usage, but existing byte[] backends wil continue to work (albeit with less efficiency than they could achieve by implementing the new backend API)

[mgravell]

@aKzenT please see updated body

[Tornhoof]

As for L1+L2 caching, you might want to talk to developers of MS FASTER https://github.com/microsoft/FASTER, which has L1+L2 support, while L2 is not strictly out of process, more like out of main memory (disk based or azure based if I remember correctly). As from my own experience with MS FASTER, it is not necessarily easy to configure properly, but covers a lot of the functionality for L1/L2.

[mgravell]

@Tornhoof aye, FASTER has come up more than a few times; because of the setup etc required, I doubt we can reasonably make that a default implementation; the most pragmatic solution there might be to provide an IDistributedCache (i.e. L2) implementation that is FASTER-based, and leave L1+L2 disabled; that would 100% be something I'd love to see done, but it doesn't need to be critical-path for this epic

[aKzenT]

@mgravell thank you a lot for the update. I'm seeing a lot of things addressed that I've missed in the past, working on multi-server web apps.

For L1+L2 Caching we have been quite happy with https://github.com/ZiggyCreatures/FusionCache in the past which is also built upon IDistributedCache. I remember that there were some features missing from IDistributedCache that made it hard to implement advanced scenarios in that library. So I would like to invite @jodydonetti to this discussion as well as he can probably best comment on these issues.

One thing I remember that was missing was being able to modify the cache entry options (i.e. life time) of a cache entry without going through a Get/Set cycle. Being able to modify the lifetime allows for some advanced scenarios like invalidating a cache entry from the client (e.g. you received a webhook notifying you about changes in data) or reducing the time to allow things like stale results.

Another thing related to cache invalidation, that is not really possible with the current API in an efficient way, is the removal/invalidation of a group of related cache entries. Let's say you have a cache for pages of a CMS system with each page being one entry. The CMS informs you about changes via a web hook, which invalidates the cache for all pages. Directly refreshing all pages might be expensive, so you would rather refresh them individually on demand. So you want to invalidate all page entries in the cache, but there is no way to get the list of entries from the cache, nor is there a good way to delete the entries. Our solution was to built this functionality ourself using a Redis Set that manages the related keys and then iterating through these keys and removing them one by one. But it felt very hacky as you cannot even use the same Redis Connection that the distributed cache uses, as far as I remember.

[mgravell]

@aKzenT re individual cache invalidation: there is a remove API that is meant to serve that function, but it doesn't allow modify of the options; I'd love to understand the need there further

re group cache invalidations: that sounds a lot like the "tag" feature of output-cache, i.e. you associate entries with zero, one or more tags, and then you can invalidate an entire tag, which nukes everything associated; the problem is: that tracking still needs to go somewhere, and it isn't necessarily an inbuilt feature of the cache backend - it was a PITA to implement reasonably on redis without breaking the cluster distribution, for example (ask me how I know!). It also isn't something that can fit in the existing IDistributedCache backend without significant work. Maybe there is some capacity there if we simplified to "zero or one" tags, but even then... I'm not sure that is something we can tackle in this epic, but I'm open to being wrong there!

[mgravell]

Re FusionCache: that isn't one I've seen before, but glancing at the homepage, I see that the fundamental design is pretty similar (some differences, but: very comparable). There is a common theme in these things - indeed, a lot of inspiration (not actual code) in the proposal draws on another implementation of the same that we had when I was at Stack Overflow (in that version, we also had some Roslyn analyzers which complained about inappropriate captured / ambient usage - very neat!). My point: lots of approaches converging around the same API shape.

[aKzenT]

@mgravell we had the same experience implementing our invalidation logic for a redis cluster setup. It's really hard to get right. I would not expect the design to provide a complete solution to this issue, but maybe there is some way that would make it possible for other libraries to support that while building on top of IDistributedCache. Maybe @jodydonetti has an idea how that could work.

As for modifying the options, in the case of FusionCache there is the possibility to allow stale entries, which are treated as expired, but still available in case the backend is not available. For these cases there is a separate timeout of how long you want to allow a result being stale. The TTL that is sent to the underlying IDistributedCache is then the initial TTL plus the additional stale timeout. So when you invalidate an entry, but still want to allow stale results, you cannot simply delete the entry. Instead you would want to update the timeout to being equal to the stale timeout. Hope that makes sense.

[mgravell]

Yep, very familiar with the idea - it is conceptually related to the "eager pre-fetch" mentioned above - with two different TTLs with slightly different meanings

[mgravell]

via twitter:

Yes, it does lack proper API there is too much redundant code we are required to write or maintain a library on our end I used the DistributedCache code snippet provided in one of the .NET blog post by you and it is definitely going to be nice to have these features.

This is referring to this repo, which offered some of the highlights of this proposal as extension methods, but without any of the "meat" that drives the additional functionality.

[aKzenT]

One thing I'm wondering is, if the choice to put the generic type parameter on the interface rather than the methods might be limitting in some cases and would require some classes to have to configure and inject multiple IDistributedCache<T> instances. I'm not sure if that is really a problem, but it would be nice to learn, why you went for that choice, which differs from other implementations that I have seen.

[mgravell]

@aKzenT fair question, and it isn't set in stone, but - reasons:

1. to allow the serializer to be injected, although that might also be possible by taking IServiceProvider
2. to allow the callback signature to be inferred in both the stateless and stateful case, although this might also be possible with a <,> method

Tell you what, I'll branch my branch and try it the other way. Although then I need a new name... dammit!

[mgravell]

@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update

I will benchmark the perf of GetService() as it applies here, but to be honest: in any reasonable L1 cache-hit scenario, I would expect the overall performance to be dominated by the deserialize; and in any L2 or backend scenario, we should anticipate the GetService() to be a rounding error compared to the L2/backend work (if it isn't: why are we caching?)

[danielmarbach]

Would it be a option to use change token for the cache key invalidation instead of the event type proposed currently?

[mgravell]

@danielmarbach in reality, I'm not sure that is feasible in this scenario; the example uses shown for that API seem pretty low-volume and low issuance frequency; file config changes etc, but if we start talking about cache: we're going to need as many change tokens as we have cached entries, and crucially: the backend layer would need to know about them; I'm mentally comparing that to how redis change notifications can work, and to do that efficiently: we don't want to store anything extra, especially at all the backend layers. Given that all we actually want/need here is the string, this seems like going a very long way out of shape, to reuse an API for the sake of it. Happy to keep considering alternatives if I'm missing something, though! Please keep it coming, that's the entire point of this!

[aKzenT]

@danielmarbach I believe it should be fairly easy to implement a GetChangeToken(string key) method as an extension method that subscribes to the event and listens to changes to the specific key. The other way arround is harder. That being said, I'm a fan of ChangeTokens and IIRC, MemoryCache uses change tokens to invalidate entries, so there might be some value to provide such an extension method directly in the framework in addition to the event @mgravell .

[normj]

To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider

[aKzenT]

@aKzenT ^^^ isn't terrible; will add notes above - we can probably go that way; I probably hadn't accounted for the improvements in generic handling in the last few C# versions (especially with delegates); in particular, I didn't have to change the usage code at all - see L107 for entirety of the usage update

I will benchmark the perf of GetService() as it applies here, but to be honest: in any reasonable L1 cache-hit scenario, I would expect the overall performance to be dominated by the deserialize; and in any L2 or backend scenario, we should anticipate the GetService() to be a rounding error compared to the L2/backend work (if it isn't: why are we caching?)

If you want to go that route, is there a particular reason why you want ICacheSerializer<T> to be generic instead of just its methods being generic? I feel like for the basic scenario of System.Text.Json, this can just be a singleton. If someone needs to differentiate serialization by type it should be easy to do using a composite style pattern.

Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example.

[aKzenT]

Instead of configuring caching and serialization by type, I would rather have the ability to have named instances that I can configure in addition to a default instance, similar to how HttpClientFactory works. FusionCache allows this by the way, if you are interested in an example.

Keyed Services might make that easy. Although I'm not sure if there is an easy way to register both a keyed cache and a keyed serializer and tell the DI system that it should use the serializer with the same key when injecting into the cache.

[niemyjski]

I never liked the name of it to start with because I might want an in memory, hybrid or distributed cache and the abstraction doesn't change. I also don't like how expiration / expiration strategy is not present (possibly it is with extension methods, I haven't looked in a long time). I also don't feel very strongly about adding generics to the interface because a cache can store all different kinds of shapes. We probably should slim down our default interface, but this is what we've found useful over the past decade: https://github.com/FoundatioFx/Foundatio#caching

[mgravell]

To add to your list of "Current Code Layout" we have an implementation at AWS for DynamoDB as backend. https://github.com/awslabs/aws-dotnet-distributed-cache-provider

Great to know, thanks! The list was never meant to be exhaustive - I'll add and clarify. The key takeaway is: we want to actively avoid breaking any implementations; the new functionality should happily work against the existing backend, with them using the additional / optional extensions if appropriate.

[unaizorrilla]

Hi

For serialization using System.Text.Json, will be a good idea to support a JsonSerializerContext. The user can set the JsonSerializerContext to be used by ICacheSerializer on some way?

[mgravell]

@unaizorrilla I'm working on serialization config currently; will keep that in mind - a good outcome might be "if you do nothing, you get context-free json; if you explicit specify the serializer, that option is available to you"

[mgravell]

from twitter

Don't you think it would be better to be able to set the expiration policy(Abs expiry/Sliding expiry) globally?

A default options object isn't unreasonable - AbsoluteExpiration obviously no-go, but AbsoluteExpirationRelativeToNow plus sliding: are usable - maybe something for TypedDistributedCacheOptions (or whatever that name becomes)

[amoerie]

@aKzenT re individual cache invalidation: there is a remove API that is meant to serve that function, but it doesn't allow modify of the options; I'd love to understand the need there further

re group cache invalidations: that sounds a lot like the "tag" feature of output-cache, i.e. you associate entries with zero, one or more tags, and then you can invalidate an entire tag, which nukes everything associated; the problem is: that tracking still needs to go somewhere, and it isn't necessarily an inbuilt feature of the cache backend - it was a PITA to implement reasonably on redis without breaking the cluster distribution, for example (ask me how I know!). It also isn't something that can fit in the existing IDistributedCache backend without significant work. Maybe there is some capacity there if we simplified to "zero or one" tags, but even then... I'm not sure that is something we can tackle in this epic, but I'm open to being wrong there!

I just wanted to add my vote for grouped cache invalidation. We have a similar manually implemented tag based bookkeeping system where each cache value is associated with a tag, and it is possible to expire all cached values for a tag. We aggressively cache some rarely updated SQL tables, but in the event they are updated, we need to evict all cached data of that table, regardless of which specific query fetched its data.

P.S. I could definitely live with the restriction that one cached value can only be associated with one tag.

[Misiu]

We also have a custom caching mechanism based on Redis and https://github.com/thepirat000/CachingFramework.Redis. We assign multiple tags to each entity or response using https://github.com/thepirat000/CachingFramework.Redis?tab=readme-ov-file#tagging-mechanism. It would be hard to live with only a signed tag per cached value because sometimes business scenarios require clearing cache per user (all cached values of a specific user) or per table.

[mgravell]

We also have a custom caching mechanism based on Redis...

Glancing at that, it sounds very similar to the multi-tag aspect of output-cache, so: I'm very familiar. The awkward bit (as you no doubt know) then becomes GC when the target keys expire, which we dealt with in output cache using sorted sets for the tag membership (so we can just ZRANGE, and we have the prior implementation to refer to). So the "how" is fine - the trickier bit becomes hooking that into the backend, since that wasn't part of IDistributedCache. But if we want to fix that: now is probably our best chance, which we could do as part of the second auxiliary backend API. I can't promise we'll add this right now, but I'm adding it to my consideration list.

[onionhammer]

Why not call GetAsync with a callback GetOrSetAsync like IMemoryCache?

One thing would be cool would be to take a page out of react-query's book; Pass an array as the key, and then use the first few items as a 'prefix' for bulk cache eviction, for example:

// get or set from the cache, fallback to database
await myCache.GetOrSetAsync([ "partner", partnerId, "users" ], () => db.GetUsersAsync(), expires: TimeSpan.FromMinutes(5));

// evict partner and everything underneath when, for example partner is deleted, i.e.
// - ["partner", "1234"]
// - ["partner", "1234", "users"]
// - ["partner", "1234", "preferences"]
await myCache.RemoveAsync(["partner", partnerId]);

[mgravell]

I haven't updated this page, but the memory-cache naming has already come up - indeed that is the plan.

Re bulk evictions: that's essentially the "tags" mentioned previously; the question here then becomes: can we introduce that metaphor cleanly since the existing backends don't have that capability? I'm still working on that question.

On Fri, 9 Feb 2024 at 21:20, Erik O'Leary @.***> wrote:

Why not call GetAsync with a callback GetOrSetAsync like IMemoryCache?

— Reply to this email directly, view it on GitHub https://github.com/dotnet/aspnetcore/issues/53255#issuecomment-1936618721 or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEHMF6YJWLJBRV2GG2YFLYS2HJZBFKMF2HI4TJMJ2XIZLTSWBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVIYTEOBUGE3TKMBTGSSG4YLNMWUWQYLTL5WGCYTFNSBKK5TBNR2WLKRVGU3TENBTGEZTAM5ENZQW2ZNJNBQXGX3MMFRGK3FMON2WE2TFMN2F65DZOBS2YSLTON2WKQ3PNVWWK3TUUZ2G64DJMNZZJAVEOR4XAZNKOJSXA33TNF2G64TZUV3GC3DVMWUDCNZWGIYDGNBXQKSHI6LQMWSWS43TOVS2K5TBNR2WLKRSGA3TGMRXGE4TMMECUR2HS4DFUVWGCYTFNSSXMYLMOVS2UMJSHA2DCNZVGAZTJAVEOR4XAZNFNRQWEZLMUV3GC3DVMWVDKNJXGI2DGMJTGAZ2O5DSNFTWOZLSUZRXEZLBORSQ . You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

-- Regards,

Marc