yihao.dai

Hello @wangqia0309, could you please provide all datanode+datacoord logs? The logs will give us more context and help us identify the root cause more quickly. Additionally, how many shards does...

> [logs.zip](https://github.com/user-attachments/files/16724490/logs.zip) log is much big, so i compress them to a zip file we don't set shard nor set partitionkey @wangqia0309 The issue seems to be related to this...

> > i just found the log like [INFO] [datacoord/services.go:1743] ["GetImportProgress done"] [jobID=452016732656013460] [resp="status:\u003c\u003e state:Importing progress:100 collection_name:\"milvus_3b_dense_test_hash\" task_progresses:\u003cfile_name:\"[milvus_3b_dense/glm_3b_msa_embeds/new_online_0820/396_002145_000000.parquet]\" file_size:472309868 progress:100 state:\"InProgress\" \u003e start_time:\"2024-08-22T10:26:13Z\" "]\n","stream":"stdout","time":"2024-08-23T05:02:22.261878478Z"} but no log contains fileStats @wangqia0309...

Submitted a PR to prevent the panic. I'll continue investigating the root cause.

Hi @wangqia0309 , Could you please clarify if there was concurrent writing to the Parquet file during the import process? During the Milvus import process, the file is read twice....

Root cause: https://github.com/apache/arrow/issues/43860

should be fixed

Benchmark: **10MB** ``` cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz BenchmarkHTTPReturn BenchmarkHTTPReturn/test_HTTPReturn BenchmarkHTTPReturn/test_HTTPReturn-12 87 13127452 ns/op 27992718 B/op 34 allocs/op BenchmarkHTTPReturn/test_HTTPReturnStream BenchmarkHTTPReturn/test_HTTPReturnStream-12 87 12804875 ns/op 14361636 B/op 31 allocs/op PASS...

rerun cpp-unit-test

/run-cpu-e2e