[Feature] Design of EventMesh Function

[xwm1992]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Feature Request

Design of EventMesh Function

For the existing EventMesh, the connector, eventmesh runtime and other modules can be jointly abstracted into a runtime component. This component can be used as connector runtime\mesh runtime\function runtime, etc. The specific component structure and interaction are shown in the following figure:

Admin

The admin component includes the task scheduling function, which can assign tasks to the connector and manage the connector life cycle. The admin service will be registered in the Registry for connector service discovery. The admin and connector use grpc to communicate.

Connector

The connector runs in the connector runtime. After the connector is started, the Admin service will be discovered from the Registry. After selecting the admin service from the available admin service list, it will report the heartbeat to the admin service and carry the currently assigned task information. There is no task information in the initial state. Admin will assign task to the connector based on the received heartbeat and task information, and the tasks will be recorded in the DB. After receiving the task, the connector performs task initialization and loads the corresponding connector plug-in.

Function

Function runs in function runtime. Function mainly provides filter/transformer/compute capabilities. The admin can configure function-related rules. Function needs to communicate through Mesh.

Mesh

Mesh runs on mesh runtime. Mesh provides routing, pub/sub, and serverless automatic expansion and contraction capabilities. Mesh runtime integrates netty server by default. All traffic will be converged and summarized in mesh. Automatic expansion and contraction can be achieved by integrating knative's serving capabilities which needs to be deployed under k8s. For k8s deployment, as shown below, different components to be run are defined through yaml. After receiving the request, the EventMesh Operator will create k8s resources for different components.

Storage

The connector can integrate a variety of storage plug-ins and uses memory storage by default.

Meta

Meta is the current meta module of eventmesh, which is used to store metadata, service discovery, etc. For specific implementation, zookeeper\nacos, etc.

EventMesh Function设计

对于现有的EventMesh来说，可以将connector、eventmesh runtime等模块共同抽象出一个runtime组件，这个组件可以作为connector runtime\mesh runtime\function runtime等等，具体的组件结构与交互如下图所示：

Admin

admin组件包含了任务调度的功能，可以对connector分配任务, 管理connector生命周期, admin将自己这个服务上报到Registry供Connector进行服务发现, admin与connector之间采用grpc方式进行通信。

Connector

connector运行于connector runtime, connector启动后会从Registry发现Admin服务，从可用的admin服务列表中选择admin服务后，向该admin服务上报心跳，并携带当前分配的任务信息，初始状态无任务信息，admin会根据收到的心跳以及任务信息来分配任务，并将任务记录到DB中。connector接收到任务后，执行任务初始化，加载对应的connector插件。

Function

function运行于function runtime, function主要提供filter/transformer/compute的能力，可以由admin来配置function相关的规则，function需通过Mesh进行通信交互。

Mesh

mesh运行于mesh runtime, mesh提供了routing、pub/sub以及serverless自动扩缩容的能力, mesh runtime默认集成了netty server, 所有的流量会收敛汇总于mesh, 通过集成knative的serving能力实现自动扩缩容, 同时需要在k8s下部署, 对于k8s部署如下图, 通过yaml定义要运行的不同组件, EventMesh Operator收到请求后会对不同组件创建k8s资源。

Storage

connector可集成多种storage插件，默认采用内存存储。

Meta

Meta即为目前eventmesh的meta模块, 用于存储元数据、服务发现等, 具体实现可使用zookeeper\nacos等。

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [ ] I agree to follow this project's Code of Conduct *