Suggesting one consumer each runtime and self-management offset mode

[iNanos]

Enhancement Request

Is your enhancement request related to a problem? Please describe

EventMesh currently chooses a consumer group-level agent, that is, there is only one real Consumer on a single EventMesh instance of a consumer group. The number of clients that a single EventMesh instance can support is limited by the number of consumers and the average number of clients in every consumer group.
Each consumer has a private resource pool. In general, this resource pool includes but is not limited to a collection of threads, such as threads used to obtain messages and consumption, and some timing threads used to access the registry. When there are a limited number of consumers After being created on the EventMesh instance, a resource explosion will occur.

The producer has simple logic, takes up less resources and can be ignored.

like this, when several consumers are created, the JVM will not have enough threads, and the memory resources will run stably

Scene P1 and Scene P2 show the number of clients that a single EventMesh instance can support in two cases (on average, a single consumer group has more clients and fewer clients).

Obviously, in terms of quantity, scenario 1 is better In scene 2.

Describe the solution you'd like

Global consumer, a single EventMesh instance has one and only one singleton Consumer & Producer, this singleton Consumer&Producer is responsible for all clients connected to the EventMesh to send and receive events, if a single EventMesh instance only has one global consumer,

include two parts

1. manage meta-data about client-group mapping ~ meta-data table like example ~ sync meta-data with broadcast

2. local ack manage ~ The consumption progress of EventMesh is managed separately from the consumption progress of the actual consumer group

Describe alternatives you've considered

[jinrongluo]

your design of using Global Consumer is great. When are you planning to implement it? @iNanos @qqeasonchen

[github-actions[bot]]

It has been 90 days since the last activity on this issue. Apache EventMesh values the voices of the community. Please don't hesitate to share your latest insights on this matter at any time, as the community is more than willing to engage in discussions regarding the development and optimization directions of this feature.

If you feel that your issue has been resolved, please feel free to close it. Should you have any additional information to share, you are welcome to reopen this issue.

[Pil0tXia]

I think this is a good design, but this issue is not urgent. The design with offset management seems a bit outdated at the moment. I will add the "discussion" label to it.