[Bug]: Milvus Stoped Index building

[akmalmasud96]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: Cluster Mode Version 2.3.20 and Standalone Mode Version 2.4.6
- Deployment mode(standalone or cluster): Both
- OS(Ubuntu or CentOS): Ubuntu

Current Behavior

I am making diskann index with 256 vector size. This issue is being faced on the cluster mode milvus and also on the standalone milvus. All the cores in index node for the cluster mode are kept in use, however, it stop indexing with the attached error. it is working fine for other collections on which I am making with diskann-indexes for the size of 512 embedding-size. What could be the issue ? What could be the issue ?

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

Milvus Cluster Mode logs

20240807 14:06:45.351478    45 ChunkManagers.cpp:112] [SEGCORE][AwsChunkManager][milvus] init AwsChunkManager with parameter[endpoint: '[s3.us-east-1.amazonaws.com:443](http://s3.us-east-1.amazonaws.com:443/)', default_bucket_name:'milvus-cluster-bucket-v2', root_path:'milvus_data', use_secure:'true']
I20240807 14:06:45.351606    45 factory.cc:20] [KNOWHERE][Create][milvus] create knowhere index DISKANN with version 1
I20240807 14:06:45.829555    45 aux_utils.cpp:1160] [KNOWHERE][build_disk_index][milvus] Using Cosine search, so need to pre-process base data into temp file. Please ensure there is additional (nd4) bytes for storing pre-processed base vectors, apart from the intermin indices and final index.
I20240807 14:06:45.831759    45 aux_utils.cpp:1190] [KNOWHERE][build_disk_index][milvus] Starting index build: R=56 L=100 Query RAM budget: 0.000149(GiB) Indexing ram budget: 24.6064(GiB)
I20240807 14:06:45.831789    45 aux_utils.cpp:1208] [KNOWHERE][build_disk_index][milvus] Compressing 256-dimensional data into 128 bytes per vector.
E20240807 14:06:45.833606    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833616    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833618    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833622    45 partition_and_pq.cpp:211] error in copy
I20240807 14:06:45.833458    45 aux_utils.cpp:1249] [KNOWHERE][build_disk_index][milvus] Generating PQ pivots
E20240807 14:06:45.833626    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833628    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833631    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833635    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833638    45 partition_and_pq.cpp:211] error in copy
E20240807 14:06:45.833649    45 partition_and_pq.cpp:211] error in copy

Standalone Mode logs,

[indexnode/task.go:516] [\"index params are ready\"] [buildID=451892472924565981] [\"index params\"=\"{\\\"build_dram_budget_gb\\\":\\\"125.129562\\\",\\\"dim\\\":\\\"256\\\",\\\"index_type\\\":\\\"DISKANN\\\",\\\"max_degree\\\":\\\"56\\\",\\\"metric_type\\\":\\\"COSINE\\\",\\\"num_build_thread\\\":\\\"32\\\",\\\"num_build_thread_ratio\\\":\\\"1\\\",\\\"pq_code_budget_gb\\\":\\\"0.000178\\\",\\\"pq_code_budget_gb_ratio\\\":\\\"0.125\\\",\\\"search_cache_budget_gb\\\":\\\"0.000143\\\",\\\"search_cache_budget_gb_ratio\\\":\\\"0.100000\\\",\\\"search_list_size\\\":\\\"100\\\"}\"]\n","stream":"stdout","time":"2024-08-17T13:32:47.067386179Z"}
{"log":"I20240817 13:32:47.067938 21587 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=warn, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]\n","stream":"stdout","time":"2024-08-17T13:32:47.068092611Z"}
{"log":"I20240817 13:32:47.070744 21587 ChunkManager.cpp:112] [SERVER][AwsChunkManager][milvus] init AwsChunkManager with parameter[endpoint=minio:9000][bucket_name=a-bucket][root_path=files][use_secure=false]\n","stream":"stdout","time":"2024-08-17T13:32:47.070844609Z"}
{"log":"I20240817 13:32:47.071208 21587 index_factory.cc:54] [KNOWHERE][Create][milvus] use key DISKANN_fp32 to create knowhere index DISKANN with version 4\n","stream":"stdout","time":"2024-08-17T13:32:47.071311724Z"}
{"log":"I20240817 13:32:47.115267 21587 aux_utils.cpp:1124] [KNOWHERE][build_disk_index][milvus] Using Cosine search, so need to pre-process base data into temp file. Please ensure there is additional (nd4) bytes for storing pre-processed base vectors, apart from the intermin indices and final index.\n","stream":"stdout","time":"2024-08-17T13:32:47.115462925Z"}
{"log":"I20240817 13:32:47.118752 21587 aux_utils.cpp:1154] [KNOWHERE][build_disk_index][milvus] Starting index build: R=56 L=100 Query RAM budget: 0.000178(GiB) Indexing ram budget: 125.13(GiB)\n","stream":"stdout","time":"2024-08-17T13:32:47.118892311Z"}
{"log":"I20240817 13:32:47.118821 21587 aux_utils.cpp:1172] [KNOWHERE][build_disk_index][milvus] Compressing 256-dimensional data into 127 bytes per vector.\n","stream":"stdout","time":"2024-08-17T13:32:47.118911734Z"}
{"log":"I20240817 13:32:47.123124 21587 aux_utils.cpp:1213] [KNOWHERE][build_disk_index][milvus] Generating PQ pivots\n","stream":"stdout","time":"2024-08-17T13:32:47.123244268Z"}
{"log":"E20240817 13:32:47.123667 21587 partition_and_pq.cpp:209] error in copy\n","stream":"stderr","time":"2024-08-17T13:32:47.123766222Z"}
{"log":"E20240817 13:32:47.123688 21587 partition_and_pq.cpp:209] error in copy\n","stream":"stderr","time":"2024-08-17T13:32:47.123781835Z"}
{"log":"E20240817 13:32:47.123693 21587 partition_and_pq.cpp:209] error in copy\n","stream":"stderr","time":"2024-08-17T13:32:47.12378496Z"}
{"log":"E20240817 13:32:47.123698 21587 partition_and_pq.cpp:209] error in copy\n","stream":"stderr","time":"2024-08-17T13:32:47.123787732Z"

Anything else?

No response