deepflow icon indicating copy to clipboard operation
deepflow copied to clipboard

[FR] can we add mq Middleware for server to save data which write to clickhouse

Open LGTH opened this issue 11 months ago • 3 comments

Search before asking

  • [X] I had searched in the issues and found no similar feature requirement.

Description

当CK的能力不足的时候,server会写入堆积导致内存增高,不然就是队列长度不够,导致数据丢弃,希望server和ck提高解耦,避免直接写入,如果server异常重启后,如流日志丢失,也是不希望出现的情况

Use case

能否增加一个MQ的中间件,避免直接写入数据到ck,主要想达到的效果是,避免ck压力过大(瞬时写入,需要削峰填谷)和server异常重启后如流日志和请求日志数据不丢失。 比如是否可以在server中增加一个写入和消费MQ的模块,用于保留流日志和请求日志

Related issues

No response

Are you willing to submit a PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

LGTH avatar Mar 24 '24 02:03 LGTH

你好,目前我们已经有做这个 feature 的规划,预计在 6.6 版本,按照版本迭代周期计算,预计时间会在下半年

1473371932 avatar Mar 26 '24 02:03 1473371932

nice

LGTH avatar Apr 01 '24 01:04 LGTH