Jiquan Long

the tracing can be found in https://github.com/milvus-io/milvus/pull/32734

reproduced on my local pc. building Trie index with 100,000 rows whose length is all 65535, the peak memory is almost 14.5GB.  And I also profiled the memory allocations...

Marisa-trie doesn't support building index streamingly, then we need to read all binlog into memory. So the peak memory will double the total binlog size now.

Sorry, I made a mistake that I use same data when trying to reproduce this issue on my local pc. If data is almost same, the trie index in fact...

We can't solve this issue until the segment can be really controlled by the `dataCoord.segment.maxSize`. By default, the max size of a segment is 1G, however, in our case, the...

Tantivy is a wal-like architechture library, and you can see details in https://fulmicoton.com/posts/behold-tantivy-part2/. The actual index thread is working at the background. In this case, indeed it will make inverted...

/assign @wangting0128 Can you try more real datasets for this?

I noticed that the goroutines and OS threads are very high compared to normal instances:  Below are the goroutines and OS threads of normal instances: 

Also, the querynode under cluster mode also encountered this issue, so maybe index building is not the root cause.

Maybe this issue is not caused by the inverted index. I noticed that there is no any inverted index building job before the Milvus disconnected from etcd. See the [logs](https://grafana-4am.zilliz.cc/explore?orgId=1&panes=%7B%22rfM%22:%7B%22datasource%22:%22vhI6Vw67k%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22vhI6Vw67k%22%7D,%22editorMode%22:%22code%22,%22expr%22:%22%7Bpod%3D%5C%22multi-vector-based-scene1-f8pw5-milvus-standalone-65cf6f86z2s9q%5C%22%7D%7C~%20%5C%22index%20params%20are%20ready%7CSuccessfully%20build%20index%5C%22%22,%22queryType%22:%22range%22,%22hide%22:false%7D,%7B%22refId%22:%22B%22,%22expr%22:%22%7Bpod%3D%5C%22multi-vector-based-scene1-f8pw5-milvus-standalone-65cf6f86z2s9q%5C%22%7D%7C~%5C%22disconnected%20from%20etcd%5C%22%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22vhI6Vw67k%22%7D,%22editorMode%22:%22code%22,%22hide%22:true%7D,%7B%22refId%22:%22C%22,%22expr%22:%22%7Bpod%3D%5C%22multi-vector-based-scene1-f8pw5-milvus-standalone-65cf6f86z2s9q%5C%22%7D%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22vhI6Vw67k%22%7D,%22editorMode%22:%22code%22,%22hide%22:true%7D%5D,%22range%22:%7B%22from%22:%221713170090742%22,%22to%22:%221713256490742%22%7D%7D%7D&schemaVersion=1).