[Feature] The filtering feature of events

[epiao55]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Feature Request

I would like to contribute a filtering feature for events. Here is my rough design idea. First, when creating a subscription, add a new json String field 'filter' to the SubscriptionItem to describe the specific filtering rules.for example

{
	"and": [{
		"field": "specversion",
		"operator": "EQ",
		"value": "1.0"
	}, {
		"or": [{
			"field": "id",
			"operator": "EQ",
			"value": "9aeb0fdf-c01e-0131-0922-9eb54906e209"
		}, {
			"field": "email",
			"operator": "CN",
			"value": "example"
		}]
	}, {
		"field": "data.contentType",
		"operator": "SW",
		"value": "image/png"
	}]
}

When a consumer consumes, check the 'filter' field and convert it into a 'FilterGroup'. Then, generate a combined predicate based on the rules and provide the filtering results. The logical relationship between the rules is described by 'and' and 'or'. The 'field' field describes the field to be checked, the 'operator' describes the specific rule, and the 'value' describes the comparison data for the rule. The value of the field field can be a nested field in the event, expressed as a hierarchical relationship using '.'. and This is the filtering process.

sequenceDiagram
    participant Consumer
    participant FilterGroup
    participant SubscriptionItem
    participant SubscriptionItem
    Consumer->>+FilterGroup: Transform event to JSON
    FilterGroup->>SubscriptionItem: Get filter rules JSON
    FilterGroup->>FilterGroup: Parse filter rules JSON
    FilterGroup->>FilterGroup: Generate combined predicate
    FilterGroup->>+FilterRule: Predicate rules
    Note over FilterRule: Loop for all rules
    FilterRule->>FilterRule: Rule 1 apply
    FilterRule->>FilterRule: Rule 2 apply
    FilterRule->>FilterRule: Rule 3 apply
    Note over FilterRule: End of loop
    FilterRule->>-FilterGroup: Allowed or denied
    alt FilterGroup allowed or denied
        FilterGroup->>-Consumer: Allowed or denied
    else FilterGroup throws exception
        FilterGroup->>Consumer: Apply default strategy
    end

And provide optional default policy to handle filtering exceptions for allowing or denying access. I have initially designed the following specific types of rules:

operator	class
EQ	EqualFilterRule
NE	NotEqualFilterRule
GT	GreaterThanFilterRule
GE	GreaterThanOrEqualFilterRule
LT	LessThanFilterRule
LE	LessThanOrEqualFilterRule
BT	BetweenFilterRule
CN	ContainsFilterRule
NC	NotContainsFilterRule
SW	StartsWithFilterRule
EW	EndsWithFilterRule
RE	RegexFilterRule
IN	InFilterRule
NIN	NotInFilterRule
EX	ExistsFilterRule
NEX	NotExistsFilterRule
NOT	NotFilterRule
NL	NullFilterRule
NN	NotNullFilterRule
EQT	EpochTimeFilterRule
GTT	GreaterThanTimeFilterRule
LTT	LessThanTimeFilterRule
BT	BetweenTimeFilterRule

I have started working on the implementation of this code. Please consider my approach, which can also be applied to similar transformation features. The bolded rules have been implemented. Rules can be continuously extended by implementing the interface.

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[xwm1992]

@epiao55 Hi, first of all, thank you for your design of the Event Filter scheme. You can refer to the code of eventmesh-filters branch. Your scheme seems to apply the event filter rule configuration and processing filtering from the consumer sdk side when subscribing. We hope to be able to use it from the EventMesh API Layer design and implement filter and transformer capabilities in EventMesh runtime. We can arrange a meeting to discuss the design plan and talk about the specific design ideas.

[epiao55]

@epiao55 Hi, first of all, thank you for your design of the Event Filter scheme. You can refer to the code of eventmesh-filters branch. Your scheme seems to apply the event filter rule configuration and processing filtering from the consumer sdk side when subscribing. We hope to be able to use it from the EventMesh API Layer design and implement filter and transformer capabilities in EventMesh runtime. We can arrange a meeting to discuss the design plan and talk about the specific design ideas. @xwm1992 Thank you for your reply, I will first understand the existing design of this part. And How can I contact you

[xwm1992]

@epiao55 Hi, first of all, thank you for your design of the Event Filter scheme. You can refer to the code of eventmesh-filters branch. Your scheme seems to apply the event filter rule configuration and processing filtering from the consumer sdk side when subscribing. We hope to be able to use it from the EventMesh API Layer design and implement filter and transformer capabilities in EventMesh runtime. We can arrange a meeting to discuss the design plan and talk about the specific design ideas. @xwm1992 Thank you for your reply, I will first understand the existing design of this part. And How can I contact you

you can contact with the eventmesh wechat helper, and this design is similar to the design of zuul 2.0. If you have a better design reference, you can also put it forward.

[qqeasonchen]

@epiao55 good idea!