kop
kop copied to clipboard
streamnative
[BUG] Kop Consume speed gradually slow by time
Describe the bug Tailing-read scenario，Kop Consume speed gradually slow by time
To Reproduce Steps to reproduce the behavior:
- use Flink to Produce message to Pulsar by Kop. Speed is about 2GB/s. Total 20 Pulsar nodes. Flink parallelism is 100.
- use another Flink job to consume Message with latest offset
- Topic total 420 partition
- initially Flink consume throughput equals produce throughtpu. Grandually consume speed slow
- finally goes into catch up read scenario (PS: it's sure consume resource is enough)
Expected behavior Because resource is enough & consume start from latest offset, Flink consume throughput should equal produce throughtput
Screenshots
You can check out the consumer related metrics to see if there is something wrong, especially about if the latency increased by time. See https://github.com/streamnative/kop/blob/master/docs/reference-metrics.md#consumer-metrics.
Besides, could you try a greater maxReadEntriesNum config?
You can check out the consumer related metrics to see if there is something wrong, especially about if the latency increased by time. See https://github.com/streamnative/kop/blob/master/docs/reference-metrics.md#consumer-metrics.
Besides, could you try a greater
maxReadEntriesNumconfig?
I will try to increase maxReadEntriesNum from 5 to 50.
Sorry i misunderstanding which metric closely related to kop consume，can you help point out ？
Sorry i misunderstanding which metric closely related to kop consume，can you help point out ？
You can use https://github.com/streamnative/kop/blob/master/docs/reference-metrics.md#request-metrics-1 to monitor the whole latency of a FETCH request.
kop_server_REQUEST_LATENCY{request="FETCH", ...}
Then you can use these metrics to see latency in different phases.
- kop_server_PREPARE_METADATA
- kop_server_MESSAGE_READ
- kop_server_FETCH_DECODE
kop_server_TOTAL_MESSAGE_READ is removed now, the docs is outdated.