pulsar-client-go icon indicating copy to clipboard operation
pulsar-client-go copied to clipboard

aynsc send [hasSpace()] control has bad performance when send large message

Open zhaoyu1989 opened this issue 3 years ago • 1 comments

Expected behavior

  1. when i send large message body,the client has a queue to hold it and send by batch, there are two triggers: 1. message count lager then maxMessages, OR 2. messge body sum is larger then 'BatchingMaxSize'
  2. Know the codes are 【bc.numMessages+1 <= bc.maxMessages && bc.buffer.ReadableBytes()+msgSize <= uint32(bc.maxBatchSize)】,and it will brings somemany Flush opertion when i send large message body

Actual behavior

  1. 【unable to add message to batch】will called when send message body size nearly 'BatchingMaxSize'

Steps to reproduce

  1. send body size lager then 3M
  2. make send fast than 1000 qps

System configuration

Pulsar version: 2.9.0

zhaoyu1989 avatar May 05 '22 15:05 zhaoyu1989

https://github.com/apache/pulsar-client-go/pull/792 might fix this problem. It guarantees at least one message to be added to a batch that will be sent.

[unable to add message to batch] is hard to produce otherwise. Because there is an internal flush already before the second Add() to batch, I believe such error only occurs when the first flush fails. Do you see any error messages prior?

zzzming avatar Jun 15 '22 21:06 zzzming