新增Nacos数据推送的反压机制 / Add new backpressure mechanism for data push Nacos.

[KomachiSion]

开源之夏是由中科院软件所“开源软件供应链点亮计划”发起并长期支持的一项暑期开源活动，旨在鼓励在校学生积极参与开源软件的开发维护，培养和发掘更多优秀的开发者，促进优秀开源软件社区的蓬勃发展，助力开源软件供应链建设。

Nacos将会作为指导组织参加本次的“开源之夏2023”

背景

随着Nacos2.0在实时性上的进步，目前Nacos的数据实时性已经可以达到秒级。在这种情况下，大规模的应用场景时，如果某个应用向Nacos订阅了大量的数据，Nacos的秒级数据推送会生成一个数据的脉冲，如果该应用的处理速率跟不上数据的推送速率，使得应用的资源被大量占用，从而导致应用的稳定性问题。此类问题也经常出现于如流式计算系统和消息系统中，比如Flink或rocketMQ。

为了解决此类问题，需要设计一个数据推送的反压机制，对Nacos的数据推送速率进行一定限制，优先保证应用系统的稳定性；在保证系统的稳定性的基础上，尽可能保证实施性要求。

同时，随着serverless等技术发展，在未来发展场景中，实时性会变的越来越重要，提供反压机制也能够让Nacos在资源充足的情况下，没有负担的进行实时性的进一步加强和提升。

反压机制作为Nacos反脆弱体系中的重要功能，其实现原理较为复杂，同学可以提前学习Flink和rocketMQ等现有产品的反压机制实现原理，并参考其实现进行方案设计，但不可以直接照搬或抄袭。

目标

为Nacos设计一个数据推送的反压机制，在客户端处理速度无法满足时，对Nacos的数据推送速率进行一定限制和降低，以优先保证客户端的稳定性。

难度

进阶

导师

邢学超 [email protected]

产出要求

• 设计Nacos反压机制的功能方案，并实现功能
• 新增Nacos反压机制功能的使用文档

能力要求

• 熟悉Java编程语言
• 熟悉常见的设计模式
• 熟悉并发编程
• 了解TCP滑动窗口
• 了解Flink、rocketMQ等产品的反压机制的原理和实现
• 熟悉markdown

---

Open Source Promotion Plan is a summer program organized by the Institute of Software Chinese Academy of Sciences and long-term supported by the Open Source Software Supply Chain Promotion Plan. It aims to encourage college students to actively participate in the maintenance and development of open source software, promote the vigorous development of open source software communities, and build the open source software supply chain together.

Nacos will join The Summer 2023 as the mentoring organization.

Background

With the improvement of real-time performance in Nacos 2.0, the real-time data in Nacos can currently reach the second level. In this case, in large-scale application scenarios, if an application subscribes to a large amount of data from Nacos, Nacos' second-level data push will generate a data pulse. If the application's processing speed cannot keep up with the data push rate, it will occupy a large amount of application resources, leading to stability issues. Such problems often occur in stream processing system and messaging system such as Flink or rocketMQ.

To solve this problem, it is necessary to design a data push backpressure mechanism for Nacos to limit and reduce the data push rate of Nacos, prioritizing the stability of the application system. On the basis of ensuring the stability of the system, the implementation requirements should be met as much as possible.

At the same time, with the development of technologies such as serverless, real-time performance will become more and more important in future development scenarios. Providing a backpressure mechanism can also allow Nacos to further strengthen and improve real-time performance without burden in the case of sufficient resources.

As an important feature of the anti-fragility system of Nacos, the implementation principle of the backpressure mechanism is relatively complex. Students can learn in advance the principles and implementation of backpressure mechanisms in existing products such as Flink and RocketMQ, and refer to their implementation for design, but cannot copy directly.

Target

Design a data push backpressure mechanism for Nacos to limit and reduce the data push rate of Nacos when the client processing speed cannot meet the requirements, prioritizing the stability of the client.

Difficulty

Advanced

Mentor

Xing Xuechao [email protected]

Output Requirements

• Design for Nacos' backpressure mechanism and implement the feature
• Add documentation for using Nacos' backpressure mechanism

Technical Requirements

• Familiar with Java programming language
• Familiar with common design patterns
• Familiar with concurrent programming
• Familiar with TCP sliding window
• Understand the principles and implementation of backpressure mechanisms in products such as Flink and RocketMQ
• Familiar with writing markdown documents

[isHuangXin]

I am interested in this issue, Can I apply to complete this project in OSPP? I will upload my solution in the proposal later.

[KomachiSion]

课题暂无同学提交合适提案， 从开源之夏2023中移除， 转化为社区任务，如果有同学或社区同学有意向计划实现，同样可以联系导师，或直接在issue中回复方案。

[KomachiSion]

I am interested in this issue, Can I apply to complete this project in OSPP? I will upload my solution in the proposal later.

This issue removed from OSPP due to no adequate proposal. If you are still interested, you can continue do it.

[godhth]

我看代码中config推送的是变化的key，client收到之后，再去主动更新。 而naming中推送的是所以信息。 请问下，这是怎么考虑的？ 能否naming 也采用推送变化key，client根据自身处理能力，量力而行，再去主动更新。 使用这种方式是否会更简单些？

I see that the key pushed by config in the code is the changed key, and the client takes the initiative to update it after receiving it. And the naming push is all information. Excuse me, how is this considered? Can naming also use push change key, client according to its own processing capability, and then take the initiative to update. Would it be easier to use this approach?

[jackie-coming]

分配给我吧，我尝试实现一下

[bs352]

@i will solve it@

Hello community, I am fairly familiar with backpressure mechanisms in flink, and would like to have a go at this issue. I tried to contact with mentor via the metioned email but got no reply. Should I post design proposals here?

[stone-98]

@i will solve it@

Hello community, I am fairly familiar with backpressure mechanisms in flink, and would like to have a go at this issue. I tried to contact with mentor via the metioned email but got no reply. Should I post design proposals here?

Can you share the design proposals here? I also want to learn.