Redis pubsub support

[rukai]

In order to solve https://github.com/shotover/shotover-proxy/issues/537 we are going to have to design a way for shotover to handle the pubsub model which goes against many of the assumptions in shotovers current design.

Issues with adding pubsub support to shotovers current design:

• Shotover does connection pooling which breaks redis per connection subscriptions
• Shotover requires each request must have exactly one response. A subscription is one request but results in many responses until the connection is closed. There are two sub problems here:
  • There is no way for the db to initiate a new message.
  • Many transforms are built around the invariant that there is exactly one response per request.
• let me know if you can think of any I missed

An initial idea I had is: Sink transforms detect subscription messages and setup a non-pooled connection that sets up subscription replies to go through Transform::transform_subscribe as described below:

Add a method to Transform for sending subscription responses backwards through the chain.

async fn transform_subscribe<'a>(&'a mut self, message_wrapper: Wrapper<'a>) -> ChainResponse {
    message_wrapper.call_next_transform_subscribe();
}

It will have its own set of invariants separate to transform. i.e. no requirement for each request to have exactly one corresponding response. That way each transform can implement handling for subscription messages if they choose but by default subscription messages are just passed through.

But this is just an initial idea and we should consider and compare lots of approaches before settling on one.

Notes on pubsub support: The client can still send messages while in subscription mode but only SUBSCRIBE, SSUBSCRIBE, SUNSUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE, PING, RESET, and QUIT are supported. If the client calls RESET then we leave subscription mode.

From looking at the implementation of redis-rs unsubscribing from all current subscriptions will also leave subscription mode.

[aembke]

For what it's worth, this is what the is_pubsub_message and related functions are for on the Frame types in redis_protocol. I had to go through a similar design process with fred, and some of the changes required to support pubsub frames made their way into the protocol library.

Feel free to ping me if you have any questions on any of this. The logic to handle the entire suite of redis response types and patterns is not trivial. Last I checked that logic required nearly 2000 LoC in fred. However, a fair amount of that logic is there only to put a 1-1 request/response facade on top of some redis interfaces that do not follow a simple 1-1 request/response pattern. Hopefully you can avoid some of these difficulties though since shotover has different use cases than a specialized client library.

I'm very interested in using shotover to poke a hole into some minikube clusters that run redis clusters, so I'm happy to lend a hand if it would help.

[benbromhead]

I've had a bit of a think about this in the past. As you've rightly identified, shotover very much sits in a request / response model (in fact the way we do async is largely predicated on this).

It might be interesting to explore generators (once they hit stable rust), as a way to shoehorn into the current req/response model. Where the response is a generator that may return 1 - N messages. -> https://doc.rust-lang.org/beta/unstable-book/language-features/generators.html.

Even with this I like @rukai idea to handle pub/sub model messages in a different method on the transform trait as it keeps invariants nice and clean.

[rukai]

RedisSinkSingle, supports pubsub as of https://github.com/shotover/shotover-proxy/pull/645

Since this issue was largely exploring the problem space and we now have a pretty good idea of what the solution looks like, I'll raise a new issue for RedisSinkCluster.