nrtsearch
nrtsearch copied to clipboard
Starting index already present on disk bugs out the S3 download
Starting index on replica from a backup when the index is already downloaded before bugs out S3 download with the log Not all bytes were read from the S3ObjectInputStream
being infinitely printed.
Command run: curl -XPOST --fail localhost:12001/v1/start_index -d "{\"indexName\": \"test_idx\",\"mode\": \"REPLICA\",\"primaryAddress\": \"localhost\",\"port\": 11001,\"restore\": {\"serviceName\": \"nrtsearch-test\",\"resourceName\": \"test_idx\"}}";
Logs:
[INFO ] 2021-06-07 09:41:29.527 [GrpcLuceneServerExecutor-4-thread-6] ArchiverImpl - Downloading resource test_idx for service nrtsearch-test version test_idx_2021-05-05_uswest2_data to directory /home/sarthakn/nrtsearch_dev/replica_files/archiver/test_idx_data/test_idx_2021-05-05_uswest2_data
[INFO ] 2021-06-07 09:41:29.564 [GrpcLuceneServerExecutor-4-thread-6] ArchiverImpl - Full object size: 39585368645
[INFO ] 2021-06-07 09:41:29.660 [GrpcLuceneServerExecutor-4-thread-6] ArchiverImpl - Object parts: 4719
[INFO ] 2021-06-07 09:41:29.940 [GrpcLuceneServerExecutor-4-thread-6] ArchiverImpl - Object streaming started...
[INFO ] 2021-06-07 09:41:29.943 [GrpcLuceneServerExecutor-4-thread-6] ArchiverImpl - Directory /home/sarthakn/nrtsearch_dev/replica_files/archiver/test_idx_data/test_idx_2021-05-05_uswest2_data already exists, not re-downloading from Archiver
Jun 07, 2021 9:41:29 AM com.amazonaws.services.s3.internal.S3AbortableInputStream close
WARNING: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
Jun 07, 2021 9:41:29 AM com.amazonaws.services.s3.internal.S3AbortableInputStream close
WARNING: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
...