Zhen Ye
Zhen Ye
Here's the bug explaination: Rocksmq Consumer's dilivery logic works as below: ``` func (c *client) consume(consumer *consumer) { ... case _, ok :=
@hung-phan - Does it keep logging forever? - Could you provide more log to debug?
milvus has multiple wal on different streamingnode，so when the streamingnode restart, milvus will recover the wal on closing streamingnode on another one, some log will be printed as expected. And...
@Classic298 If your log content is ``` wal is on shutdown ``` It already be fixed by #45154, will be released at 2.6.6.
Checking the log from recovery storage and flowgraph. All data has already been persisted into wal. ``` > grep 'insert entity' pulsar_lost.log | awk '{match($0, "rows=([0-9]+)", a); b+=a[1]}END{print b}' 30000000...
Out retention duration of pulsar is 12h. The tailing data of wal before restarting pulsar is deleted by pulsar. So after restarting streamingnode and pulsar broker, the data is lost...
pr #42653 has been produced two solution for user. see the description of `backlogAutoClearBytes` . However, it only protect the data path of milvus, doesn't promise at query path. We...
Verify the milvus protection without pulsar retention ``` pulsar： backlogQuotaDefaultLimitBytes=-1 [backlogQuotaDefaultLimitGB](https://pulsar.apache.org/reference/#/next/config/reference-configuration-broker?id=backlogquotadefaultlimitgb)=-1 [subscriptionExpirationTimeMinutes](https://pulsar.apache.org/reference/#/next/config/reference-configuration-broker?id=subscriptionexpirationtimeminutes)=0 [defaultRetentionTimeInMinutes](https://pulsar.apache.org/reference/#/next/config/reference-configuration-broker?id=defaultretentiontimeinminutes)=30 ``` ``` milvus: pulsar.backlogAutoClearBytes=0 streaming.walTruncate.sampleInterval=1m streaming.walTruncate.retentionInterval=10m ``` Write some test case into wal without flush. stop milvus...