Support Redis server-assisted client side caching (new in Redis 6)

[giggio]

I saw the comment from @mgravell on https://github.com/StackExchange/StackExchange.Redis/issues/1459#issuecomment-627813702, so I thought it would be nice to have a separate issue dedicated to it.

More info at: https://redis.io/topics/client-side-caching

[bhalval]

Hello Marc and Team, We are new to Redis Cache and our team is exploring the StackExchange.Redis for our C# and .Net Framework based application. We are mostly able to interface with Redis Cache using the StackExchange.Redis client but one thing we are still struggling with is https://redis.io/topics/client-side-caching. How to do this using StackExchange.Redis client? OR It is Redis server management topic and should be handled at server side? If yes then how to do it on server side? Hoping for a quick response, thanks in advance.

Regards, Dheeraj Bhalval

[KamranShahid]

@bhalval Check https://github.com/StackExchange/StackExchange.Redis/issues/1459 Client caching is a very new feature so obviously take time.

[mgravell]

I can confirm this isn't something we've had chance to look into yet, but it is on our roadmap - and as it happens, I was rewriting our own internal pre-redis-6 implementation of this just last week (for Stack Overflow). Properly designing, implementing, testing, documenting etc this is going to take a little while.

[bhalval]

@mgravell and @KamranShahid Thanks a lot for confirming as I was fretting over it for past week. So as an alternative, what would be your suggestion for client side caching?

[mgravell]

I do not have a specific suggestion there for now. As I say: it is on our list to look at. For other things for now: you'd have to look. We have some internal code we've wanted to open up for a while, but it needs time and effort.

On Sun, 14 Jun 2020, 17:32 bhalval, [email protected] wrote:

@mgravell https://github.com/mgravell and @KamranShahid https://github.com/KamranShahid Thanks a lot for confirming as I was fretting over it for past week. So as an alternative, what would be your suggestion for client side caching?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/StackExchange/StackExchange.Redis/issues/1461#issuecomment-643790243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEHMDK47FK67SMHKCI5Y3RWT3Q5ANCNFSM4NABAEAA .

[bhalval]

Thank you @mgravell.

[haiduc32]

Are there any updates? Has anyone found a way to get it working? Any workarounds?

I tried to Get Client Id (by guessing mostly) on the Subscription connection and set tracking with redirect, but it never receives invalidate messages (or maybe fails silently). I also tried ServiceStack.Redis, but it just throws some errors when I try to use it with tracking.

[Greatsamps]

Would also be interested in this

[BK-Soft]

@mgravell why not open the internal code for this feature? (it seems a good feature that could boost Redis )

[mgravell]

Because everything requires time and effort, and the more code we add, the more code we need to support. The library is massively downloaded, and we get zero paid hours to support it. Math.

On Fri, 22 Jan 2021, 21:50 BRAHIM Kamel, [email protected] wrote:

@mgravell https://github.com/mgravell why not open the internal code for this feature? (it seems a good feature that could boost Redis )

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/StackExchange/StackExchange.Redis/issues/1461#issuecomment-765704701, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEHMHYSSQQO47DZXXN2FLS3HXLTANCNFSM4NABAEAA .

[BK-Soft]

@mgravell all your arguments are understandable and you're doing a great job, but opening the code to the community (even with a fork and marking it as experimental) could reduce the time effort from maybe days/months to a few hours. right?

[KamranShahid]

@mgravell all your arguments are understandable and you're doing a great job, but opening the code to the community (even with a fork and marking it as experimental) could reduce the time effort from maybe days/months to a few hours. right?

isn't code available here? i think you can download it can play with whatever you can try

[BK-Soft]

@KamranShahid it's an internal code as mentioned by @mgravell there is no way to play with it

[KamranShahid]

@KamranShahid it's an internal code as mentioned by @mgravell there is no way to play with it

Ok. is it also obfuscated :)

[br3nt]

It seems we could use var response = db.Execute("CLIENT", "TRACKING", "ON"); to enable client tracking, but how do we then "listen" for the corresponding invalidation messages?

[br3nt]

I have managed to enable client tracking manually using Execute but I have run into a road block.

StackExchange.Redis seems to create two connections: one for the database, and one for the subscriber.

So, var invalidatorClientId = invalidatorDb.Execute("CLIENT", "ID"); gets the client id for the database connection rather than the subscriber connection.

This means that clientDb1.ExecuteAsync("CLIENT", "TRACKING", "ON", "REDIRECT", (string)invalidatorClientId) is setting up redirection to the incorrect connection.

Does anyone know of a way to get the client id of the subscriber connection??

Just for reference, this is my current code... Im setting up client tracking with redirection because I assume RESP2 is used by default by StackExchange.Redis.

// connection 1 - used to receive invalidation messages
var invalidator = ConnectionMultiplexer.Connect("localhost:6379", Console.Out);
var invalidatorDb = invalidator.GetDatabase();
var invalidatorClientId = invalidatorDb.Execute("CLIENT", "ID");

// connection 2 - used to test client tracking
var client1 = ConnectionMultiplexer.Connect("localhost:6379", Console.Out);
var clientDb1 = client1.GetDatabase();
var clientId1 = clientDb1.Execute("CLIENT", "ID");

// connection 3 - used to test client tracking
var client2 = ConnectionMultiplexer.Connect("localhost:6379", Console.Out);
var clientDb2 = client1.GetDatabase();
var clientId2 = clientDb1.Execute("CLIENT", "ID");

// subsribe to invalidation messages
var subscriber = invalidator.GetSubscriber();
subscriber.Subscribe("__redis__:invalidate").OnMessage(message =>
{
    Console.WriteLine($"__redis__:invalidate: {message}");
});

// enable client tracking
Console.Write("clientDb1 tracking: ");
Console.WriteLine(await clientDb1.ExecuteAsync("CLIENT", "TRACKING", "ON", "REDIRECT", (string)invalidatorClientId));

Console.Write("clientDb2 tracking: ");
Console.WriteLine(await clientDb2.ExecuteAsync("CLIENT", "TRACKING", "ON", "REDIRECT", (string)invalidatorClientId));

// register keys to enable tracking
await clientDb1.StringSetAsync("a", "Hi");
await clientDb2.StringSetAsync("a", "Hey");

// test client tracking by updating of the keys
// it is expected that messages will be published by the server to the `__redis__:invalidate` channel
// the subscriber handler above, should print the invalidation message to the console
int i = 0;
while (!stoppingToken.IsCancellationRequested)
{
    clientDb1.StringSet("a", $"A{i++}");
    Console.WriteLine(clientDb2.StringGet("a"));

    //_logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
    await Task.Delay(1000, stoppingToken);
}

[mgravell]

As discussed in our Stack Overflow conversation here, this will require library changes, to (reliably) get the CLIENT ID as part of the initial connection handshake. They're not huge or risky changes, note.

[kotovaleksandr]

@br3nt Thanks for your example! I'm trying too create client cache over StackExchange.Redis. Problem not only in client id for subscriber connection. If sent from redis-cli cache invalidation message to application with "redis:invalidate" subscriber - connection will be broken with error: Cannot convert to RedisValue: MultiBulk.

127.0.0.1:6379> CLIENT TRACKING on redirect 207 OK 127.0.0.1:6379> get 12 "666" 127.0.0.1:6379> set 12 666 OK 127.0.0.1:6379>

StackExchange.Redis.dll!StackExchange.Redis.RawResult.AsRedisValue() Line 173 StackExchange.Redis.dll!StackExchange.Redis.PhysicalConnection.MatchResult(StackExchange.Redis.RawResult result) Line 1408 StackExchange.Redis.dll!StackExchange.Redis.PhysicalConnection.ProcessBuffer(ref System.Buffers.ReadOnlySequence buffer) Line 1576 StackExchange.Redis.dll!StackExchange.Redis.PhysicalConnection.ReadFromPipe() Line 1507

This is confusing, I don't know of an alternative way to read the raw data from pub.

[kewur]

Is this in the roadmap for 2021?

[MartyBJones]

I am also looking for this functionality. Any idea on when it will be implemented?

[AlthalusDGr8]

Has there been any updates on this feature?

[b-twis]

Would love to see this feature included as it would greatly reduce the complexity of cache invalidation handling we have now.

[MartyBJones]

Nick, is there an ETA on when version 3 will be released? I see that you marked this feature for version 3.

[joshsten]

We'd like this too.

[KamranShahid]

Nick, is there an ETA on when version 3 will be released? I see that you marked this feature for version 3.

For open source project there is generally no ETA. It depends on free time of involved developers. You can yourself contribute for it's development

[GT185076]

I have managed to enable client tracking manually using Execute but I have run into a road block.

StackExchange.Redis seems to create two connections: one for the database, and one for the subscriber.

So, var invalidatorClientId = invalidatorDb.Execute("CLIENT", "ID"); gets the client id for the database connection rather than the subscriber connection.

This means that clientDb1.ExecuteAsync("CLIENT", "TRACKING", "ON", "REDIRECT", (string)invalidatorClientId) is setting up redirection to the incorrect connection.

Does anyone know of a way to get the client id of the subscriber connection??

Just for reference, this is my current code... Im setting up client tracking with redirection because I assume RESP2 is used by default by StackExchange.Redis.

// connection 1 - used to receive invalidation messages
var invalidator = ConnectionMultiplexer.Connect("localhost:6379", Console.Out);
var invalidatorDb = invalidator.GetDatabase();
var invalidatorClientId = invalidatorDb.Execute("CLIENT", "ID");

// connection 2 - used to test client tracking
var client1 = ConnectionMultiplexer.Connect("localhost:6379", Console.Out);
var clientDb1 = client1.GetDatabase();
var clientId1 = clientDb1.Execute("CLIENT", "ID");

// connection 3 - used to test client tracking
var client2 = ConnectionMultiplexer.Connect("localhost:6379", Console.Out);
var clientDb2 = client1.GetDatabase();
var clientId2 = clientDb1.Execute("CLIENT", "ID");

// subsribe to invalidation messages
var subscriber = invalidator.GetSubscriber();
subscriber.Subscribe("__redis__:invalidate").OnMessage(message =>
{
    Console.WriteLine($"__redis__:invalidate: {message}");
});

// enable client tracking
Console.Write("clientDb1 tracking: ");
Console.WriteLine(await clientDb1.ExecuteAsync("CLIENT", "TRACKING", "ON", "REDIRECT", (string)invalidatorClientId));

Console.Write("clientDb2 tracking: ");
Console.WriteLine(await clientDb2.ExecuteAsync("CLIENT", "TRACKING", "ON", "REDIRECT", (string)invalidatorClientId));

// register keys to enable tracking
await clientDb1.StringSetAsync("a", "Hi");
await clientDb2.StringSetAsync("a", "Hey");

// test client tracking by updating of the keys
// it is expected that messages will be published by the server to the `__redis__:invalidate` channel
// the subscriber handler above, should print the invalidation message to the console
int i = 0;
while (!stoppingToken.IsCancellationRequested)
{
    clientDb1.StringSet("a", $"A{i++}");
    Console.WriteLine(clientDb2.StringGet("a"));

    //_logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
    await Task.Delay(1000, stoppingToken);
}

hey , it's seem that client tracking on redirect doesnt work when the client redirection is on the stackexchange api . it's work fine in redis-cli why ?

[GT185076]

I found the problem, there is a bug here when calling this stack Exchange method on subscribed clients.

internal RedisValue AsRedisValue() { if (IsNull) return RedisValue.Null; switch (Type) { case ResultType.Integer: long i64; if (TryGetInt64(out i64)) return (RedisValue)i64; break; case ResultType.SimpleString: case ResultType.BulkString: return (RedisValue)GetBlob(); } throw new InvalidCastException("Cannot convert to RedisValue: " + Type); }

I got this : System.InvalidCastException: 'Cannot convert to RedisValue: MultiBulk' ..

The problem is that I don't care about the value - I need the key ! - subscribe client should not call this method and not raise exception at all.

[mgravell]

I wouldn't categorize that as a bug; simply: it isn't the right type for whatever is happening, which seems to be multiple values coming back into something being cast as RedisValue (which represents a single value). So: which operation does this RedisValue relate to? It isn't obvious from the code shown.

There's a reason we don't currently expose this feature in the library: it has not yet been investigated, designed, implemented, tested, documented, or supported :)

[rsabirov]

Any thoughts on when this can be implemented? This seems to be a killer feature for all the apps using Redis for caching.

Maybe we can create a Book of work on what should be implemented in the library to support this?

[rsabirov]

I did some experiments on client-side caching.

For prototyping, I used the following workaround to get a subscription client id using CLIENT LIST command

        public void StackExchangeLibraryWorkaround()
        {
            var options = ConfigurationOptions.Parse("localhost,allowAdmin=true");
            var clientName = "client1";
            var configurationOptions = options.Apply(_ => _.ClientName = clientName );
            
            var connection1 = ConnectionMultiplexer.Connect(configurationOptions, Console.Out);

            connection1.ConnectionRestored += (sender, args) =>
            {
                //TODO resubscribe here once both Interactive and Subscription connections are established
            };
            
            var subscriber1 = connection1.GetSubscriber();
            subscriber1.Subscribe("__redis__:invalidate").OnMessage(message =>
            {
                var key = message.Message.ToString();
                Console.WriteLine($"Key '{key}' changed");
                //TODO: process cache invalidation messages
            });

            var db1 = connection1.GetDatabase();

            var clientList = connection1.GetServer(connection1.GetEndPoints().First()).ClientList();
            var subscriptionClient = clientList.FirstOrDefault(c => c.Name == clientName  && c.SubscriptionCount > 0);
            // turn on client side validation messages for subscription client
            db1.Execute("CLIENT", "TRACKING", "ON", "REDIRECT", subscriptionClient.Id.ToString());

            // get value for the key and "subscribe" for the changes
            db1.StringGet("foo");


            // second client connecting and changing the value of the key
            var connection2 = ConnectionMultiplexer.Connect(options.Apply(_ => _.ClientName = "client2"), Console.Out);
            var db2 = connection2.GetDatabase();
            db2.StringSet("foo", "buzz3");

            Thread.Sleep(5000);
        }

Even if we will implement getting the subscription client id internally there is a problem with "2 connections/clients" implementation.

It will be very hard to have an invariant "Client-side caching enabled on connection is before we send any use fired command". Even if we could somehow implement a happy path for the initial multiplexor connection, it would be very hard to implement reliable reconnection handling for all possible cases. For example:

• if we lose connectivity on the Subscription connection only, we should not send any commands on the main connection while we will not reconnect the Subscription channel back again, then we should get the connection ID and send CLIENT TRACKING ON...

I think the easiest and most practical would be to implement a single connection/client model for client-side caching.

@mgravell what was the initial reason for having a separate connection for subscriptions?