Pipeline support multiple protocols

[localvar]

Background

Easegress leverages pipeline for traffic orchestration, but the current pipeline implementation only supports HTTP protocol. Although we have added the support of other protocols like MQTT, TCP/UDP with a workaround solution, it is very hard(if not impossible) to reuse filters originally designed for the HTTP protocol in these protocols.

Objective

Redesign/refactor the existing pipeline implementation, make it possible to support other protocols, and make it possible to reuse one type of filter in different protocols.

[suchen-sci]

Design

Server (http, mqtt, tcp) create context (http context, mqtt context, tcp context) based on single request, and put context into pipeline which is a chain of filters. Filters exec one by one by using given context.

Context

Context is highly related to protocol. For example, HTTPContext may contain Request and Response fields, and MQTTContext may contain Broker, Client, Topic and Payload. So, when we support a new protocol, we need to create a corresponding Context carefully.

Context:

• HTTPContext
• MQTTContext
  • long connection
  • Broker, Client is necessary to manage status (only packets are not enough)
• TCPContext

Filters

Pipeline is a chain of filters. Each filter do certain things to context.

Filter:

• HTTPFilter
  • HandleHTTP(context.HTTPContext)
• MQTTFilter
  • HandleMQTT(context.MQTTContext)
• TCPFilter
  • HandleTCP(context.TCPContext)

Pipeline:

Pipeline:

• Protocol() -> protocol of current pipeline
• Handle(Context) call filter Handle function based on protocol.
• Pipeline if general and protocol unrelated since there are only control

Server

• http -> HTTPServer
• mqtt -> MQTTProxy
• tcp -> TCPProxy

Details for MQTT:
1. mqtt client connected
2. mqtt client publish message -> mqtt broker -> pipeline -> Kafka

Note: later we may change Kafka part into a filter and topic mapping part into a filter. 

This is not the final design, the design may change during coding.

[J1031]

I think it's better to treat multiple protocols as Input and Output. Different parts of the specific protocol is just some props of the one. And it should support multiple protocols at the same time, that is exposing the same logic to multiple protocols.

[suchen-sci]

Hi, thanks a lot for your advice! The Pipeline itself supports multiple protocols at same time, but we need to design protocol specific context and filters. For example, HTTPContext need request and response, but MQTTContext may need client field since it is a long connection. For treating multiple protocols as Input and Output, I don't really understand what do you mean, can you give us more information or some basic example? Currently, our design of Pipeline follows original HTTPPipeline but add extra protocol filed in yaml file.

[J1031]

Thanks for your reply.
What I mean is that I think it's better to have an Protocol Layer.The layer handles protocol specific things and makes Pipeline protocol independent. I copied a pipeline example from the cookbook and changed HTTPPipeline to Pipeline

name: pipeline-api
kind: Pipeline
flow:
  - filter: agg
filters:
  - name: agg
    kind: APIAggregator
    pipelines:
      - name: pipeline-demo
      - name: pipeline-demo1
      - name: pipeline-demo2 

And then define multiple protocols as the following

kind: XXX
name: protocols-demo
protocols:
 - protocol: http
 - protocol: tcp
flow:
  - filter: f1
filters:
  - name: f1
    kind: APIAggregator
    pipelines:
      - name: pipeline-api

The Pipeline itself supports multiple protocols at same time,

I can't find any exmaples.Can you show me some examples? Thanks

[suchen-sci]

Hi, thanks a lot for your example! This feature is still work in progress, but when it finished, we will update documents and examples for it.

Thanks a lot for your design. If I don't understand it wrong, your design means that for all filters in Pipeline pipeline-api actually support multiple protocols, that means APIAggregator agg support HTTP, MQTT, TCP protocols and have same parameters for these protocols to work. But in practice, every server may have their own unique pipeline with specific filters and parameters. Even two http pipelines may have different filters and different parameters. Although this feature is still in progress, the design look like this:

name: pipeline-demo
kind: Pipeline
protocol: HTTP
flow:
  - filter: header-validator
  - filter: proxy
filters:
  - kind: Validator
    name: header-validator
    headers:
      Is-Valid:
        values: ["abc", "goodplan"]
        regexp: "^ok-.+$"
  - name: proxy
    kind: Proxy

In this case, we can have protocol specific filter (some filters may support multiple protocols, some filters may only support one or two protocols). And pipeline will check if the filter supports the given protocol.

[J1031]

nice! I will the it when the feature is released.

[suchen-sci]

#352 MQTT Pipeline Develop Guide #352 contain information about MQTT pipeline and filter development. Include example yaml.