Possible refactor: Make Packet Filtering + Content Routing separate traits

[markmandel]

It has come up in discussion that it might be optimal to have separate traits for Content Filtering/Modifications, and also for Content Routing.

Right now, they are a single Filter trait which can do both, but it could be split in two.

This ticket exists to keep track of this design choice, and see if it is applicable as we build out more usable filters and cover more use cases.

Question

Does Envoy/xDS have this as a concept already?

Context:

2. Whether routing should be a separate API/config layer from filtering. On this, I'm not sure. On one side, I like how we have one interface for modification. This is nice from a perspective of utilising and writing Filters -- there's less to learn, and configuration seems like it should be simpler. I can also see cases (such at this one) wherein being able to manipulate packets AND route at the same time is very useful. That being said, having a separation of the Filters and Routers (wondering if Envoy has a name for this already) might lead to more flexible implementations that are nicely decoupled. My take for now -- let's write up a ticket with the question/sacrificial design of what it would look like with a split. As we get deeper into creating more filters, we can go back that ticket and see if it makes sense as we get more experience with the system. I'd hate to add complexity before we really feel we need it, or go down a path of premature optimisation. How does that sound?

Originally posted by @markmandel in https://github.com/googleforgames/quilkin/pull/98#discussion_r486668487

[XAMPPRocky]

I wonder if this could be solved instead of needing to split filtering from routing, what if instead we had a different way to execute filters to achieve the same effect. Because I think the thing that is interesting about this separation is that you could execute some filtering out of order.

I was thinking in addition to FilterChain, we could have FilterSet (This could also be added to FilterChain but for the sake of discussion assume a separate type), which would allow out-of-order execution of filters, that terminates on the None. The benefit of this, is that we leave it up to the user to decide what they want to execute out-of-order, so that they can configure it to match their business logic. I'm imagining something like the following.

// Executed in-order
FilterChain([
    ValidatePacket,
    // These three are executed out-of-order
    FilterSet([
        RateLimit,
        LoadBalancer,
        BlockList,
    ]),
    // Once the set is completed, the chain continues
    Compress,
    Encrypt,
])

[XAMPPRocky]

I think we should revisit this, as routing using filters has already reached some limitations. For example, putting a load balancer filter in front of token router filter can drop the endpoint with the token required to route the packet, and lead to packet loss. Adding more state to routing filters only makes this more complicated and more potential foot guns for users.

Instead we should separate routing from filters entirely, I don't know if we want a Router trait as originally proposed. I think for now we just want to support IP routing and token routing, which could supported in a single enum. So the change would be after the filter chain has run (could also be before if we want to strip the token early), the proxy's Router decides whether to send it to a single IP or checks for a token and routes based on that.

enum Router {
   Ip(SocketAddr),
   LoadBalance(Vec<SocketAddr>),
   Tokens(HashMap<Vec<u8>, Vec<SocketAddr>>),
}