zilliztech
[Bug]: backup fail with 'Get empty input path, but segment should not be empty'
Current Behavior
when i use milvus-backup create -n large_backup20250214 -c LLM_MANAGER_LARGE
2025/02/14 09:16:22.574 +00:00] [INFO] [core/backup_impl_create_backup.go:620] ["collections to backup"] [collections="[LLM_MANAGER_LARGE]"] [2025/02/14 09:16:22.574 +00:00] [INFO] [core/backup_impl_create_backup.go:218] ["start backup collection"] [db=default] [collection=LLM_MANAGER_LARGE] [2025/02/14 09:16:22.582 +00:00] [INFO] [core/backup_impl_create_backup.go:251] ["try to get index"] [collection_name=LLM_MANAGER_LARGE] [2025/02/14 09:16:22.596 +00:00] [INFO] [core/backup_impl_create_backup.go:271] ["field index"] [collection_name=LLM_MANAGER_LARGE] [field_name=dataId] ["index info"="[]"] [2025/02/14 09:16:22.617 +00:00] [INFO] [core/backup_impl_create_backup.go:271] ["field index"] [collection_name=LLM_MANAGER_LARGE] [field_name=embedding] ["index info"="[{}]"] [2025/02/14 09:16:22.630 +00:00] [INFO] [core/backup_impl_create_backup.go:271] ["field index"] [collection_name=LLM_MANAGER_LARGE] [field_name=query] ["index info"="[]"] [2025/02/14 09:16:27.577 +00:00] [INFO] [core/backup_impl_create_backup.go:357] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=default] [collectionName=LLM_MANAGER_LARGE] [segmentNumBeforeFlush=993] [2025/02/14 09:21:16.382 +00:00] [ERROR] [core/backup_impl_create_backup.go:363] ["fail to flush the collection"] [databaseName=default] [collectionName=LLM_MANAGER_LARGE] [error="channel not found[channel=by-dev-rootcoord-dml_4_451680931584957868v0]"] [errorVerbose="channel not found[channel=by-dev-rootcoord-dml_4_451680931584957868v0]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/[email protected]/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).FlushV2\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/[email protected]/client/insert.go:219\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).FlushV2\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:92\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).backupCollectionPrepare\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:361\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func1.1\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:629\n | github.com/zilliztech/milvus-backup/internal/util/retry.Do\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:628\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) channel not found[channel=by-dev-rootcoord-dml_4_451680931584957868v0]\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).backupCollectionPrepare\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:363\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func1.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:629\ngithub.com/zilliztech/milvus-backup/internal/util/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:628\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
when i use milvus-backup create -n large_backup20250214 -c LLM_MANAGER_LARGE -f
[2025/02/14 09:30:29.301 +00:00] [INFO] [core/backup_impl_create_backup.go:495] ["finish build partition info"] [collectionName=LLM_MANAGER_LARGE] [partitionNum=0] [2025/02/14 09:30:29.301 +00:00] [INFO] [core/backup_impl_create_backup.go:650] ["Finish flush all collections"] [2025/02/14 09:30:29.301 +00:00] [INFO] [core/backup_impl_create_backup.go:655] ["before backupCollectionExecute"] [collection=LLM_MANAGER_LARGE] [2025/02/14 09:30:29.302 +00:00] [INFO] [core/backup_impl_create_backup.go:510] [backupCollectionExecute] [collectionMeta="id:"51883ec2-eab6-11ef-a82e-0242ac120005" start_time:1739525424 collection_id:451680931584957868 db_name:"default" collection_name:"LLM_MANAGER_LARGE" schema:<name:"LLM_MANAGER_LARGE" description:"collection description" fields:<fieldID:100 name:"dataId" is_primary_key:true description:"INT" data_type:Int64 > fields:<fieldID:101 name:"embedding" description:"float vector" data_type:FloatVector type_params:<key:"dim" value:"1024" > > fields:<fieldID:102 name:"query" description:"VarChar" data_type:VarChar type_params:<key:"max_length" value:"3000" > > > shards_num:2 has_index:true index_infos:<field_name:"embedding" index_name:"default" index_type:"IVF_FLAT" params:<key:"index_type" value:"IVF_FLAT" > params:<key:"metric_type" value:"IP" > params:<key:"params" value:"{\"nlist\":\"128\"}" > > load_state:"Loading" "] [2025/02/14 09:30:29.302 +00:00] [INFO] [core/backup_impl_create_backup.go:517] ["Begin copy data"] [dbName=default] [collectionName=LLM_MANAGER_LARGE] [segmentNum=993] [2025/02/14 09:30:30.234 +00:00] [ERROR] [core/backup_impl_create_backup.go:525] ["Fail to fill segment backup info"] [collection_id=451680931584957868] [partition_id=451680931591985955] [segment_id=451680931639572005] [group_id=0] [error="Get empty input path, but segment should not be empty, file/insert_log/451680931584957868/451680931591985955/451680931639572005/"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).backupCollectionExecute\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:525\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:657\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"] [2025/02/14 09:30:30.235 +00:00] [WARN] [core/backup_impl_create_backup.go:568] ["Resume Milvus GC Error:This warn won't fail the backup process. Pause GC can protect data not to be GCed during backup, it is necessary to backup very large data(cost more than a hour)."] [error="Get "http://localhost:9091/management/datacoord/garbage_collection/resume": dial tcp 127.0.0.1:9091: connect: connection refused"] Get empty input path, but segment should not be empty, file/insert_log/451680931584957868/451680931591985955/451680931639572005/
Steps To Reproduce
as above
Environment
distributed milvus version of 2.4.3 milvus-backup version of 0.4.13
Config
level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
console: true # whether print log to console
file:
rootPath: "logs/backup.log"
http:
simpleResponse: true
# milvus proxy address, compatible to milvus.yaml
milvus:
address: 172.17.132.197
port: 31530
authorizationEnabled: false
# tls mode values [0, 1, 2]
# 0 is close, 1 is one-way authentication, 2 is two-way authentication.
tlsMode: 0
minio:
# cloudProvider: "minio" # deprecated use storageType instead
address: 172.17.132.197 # Address of MinIO/S3
port: 30900 # Port of MinIO/S3
accessKeyID: minioadmin # accessKeyID of MinIO/S3
secretAccessKey: minioadmin # MinIO/S3 encryption string
useSSL: false # Access to MinIO/S3 with SSL
useIAM: false
iamEndpoint: ""
bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
rootPath: "file" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance
# only for azure
backupAccessKeyID: minioadmin # accessKeyID of MinIO/S3
backupSecretAccessKey: minioadmin # MinIO/S3 encryption string
# thread pool to copy data. reduce it if blocks your storage's network bandwidth
copydata: 128
# Collection level parallelism to restore
restoreCollection: 2
# keep temporary files during restore, only use to debug
keepTempFiles: false
# Pause GC during backup through Milvus Http API.
gcPause:
enable: true
seconds: 7200
address: http://localhost:9091
Anything else?
It is not the first time I use milvus-backup.About half year ago, with the same enviroment, I backup and restore this collection suceessfully.
This collection with increasing data series(currently 3M) has been serving for more than half year. It was first restored from a 2.2.6 standalone milvus.
After I unload this collection, I can not load it anymore after a few tries.
Use minio client to check whether this path exists in the bucket ""milvus-bucket"": file/insert_log/451680931584957868/451680931591985955/451680931639572005
Use minio client to check whether this path exists in the bucket ""milvus-bucket"": file/insert_log/451680931584957868/451680931591985955/451680931639572005
no such key
Please try the latest version 0.5.1, which contains many fixes.
Does the latest version support milvus 2.4.1 and 2.2.6? Previously these is a version compatible talbe, but now it is disappeared.
Please try the latest version 0.5.1, which contains many fixes.
Using the latest version, create succeeded but restore caused error. milvus-backup restore -c LLM_MANAGER_LARGE -n large_backup20250217 -s _20250217
[2025/02/17 09:51:13.415 +00:00] [ERROR] [core/backup_impl_restore_backup.go:378] ["executeRestoreCollectionTask failed"] [TargetDBName=default] [TargetCollectionName=LLM_MANAGER_LARGE_20250217] [error="restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore L0 segment: restore_collection: restore L0 segment bulk insert via grpc: restore_collection: failed to bulk insert via grpc: client: bulk insert failed: client: operation failed: error_code:UnexpectedError reason:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed" code:2100 detail:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed""] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:378\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:71\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"] [2025/02/17 09:51:13.415 +00:00] [ERROR] [core/backup_impl_restore_backup.go:337] ["execute restore collection fail"] [backupId=b4b3cabf-ed11-11ef-9315-0242ac12000d] [error="workerpool: execute job restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore L0 segment: restore_collection: restore L0 segment bulk insert via grpc: restore_collection: failed to bulk insert via grpc: client: bulk insert failed: client: operation failed: error_code:UnexpectedError reason:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed" code:2100 detail:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed""] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:337\ngithub.com/zilliztech/milvus-backup/cmd.init.func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:85\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:989\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:19\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.5/x64/src/runtime/proc.go:272"] workerpool: execute job restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore L0 segment: restore_collection: restore L0 segment bulk insert via grpc: restore_collection: failed to bulk insert via grpc: client: bulk insert failed: client: operation failed: error_code:UnexpectedError reason:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed" code:2100 detail:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed"
Please try the latest version 0.5.1, which contains many fixes.
Using the latest version, create succeeded but restore caused error. milvus-backup restore -c LLM_MANAGER_LARGE -n large_backup20250217 -s _20250217
[2025/02/17 09:51:13.415 +00:00] [ERROR] [core/backup_impl_restore_backup.go:378] ["executeRestoreCollectionTask failed"] [TargetDBName=default] [TargetCollectionName=LLM_MANAGER_LARGE_20250217] [error="restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore L0 segment: restore_collection: restore L0 segment bulk insert via grpc: restore_collection: failed to bulk insert via grpc: client: bulk insert failed: client: operation failed: error_code:UnexpectedError reason:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed" code:2100 detail:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed""] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:378\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:71\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"] [2025/02/17 09:51:13.415 +00:00] [ERROR] [core/backup_impl_restore_backup.go:337] ["execute restore collection fail"] [backupId=b4b3cabf-ed11-11ef-9315-0242ac12000d] [error="workerpool: execute job restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore L0 segment: restore_collection: restore L0 segment bulk insert via grpc: restore_collection: failed to bulk insert via grpc: client: bulk insert failed: client: operation failed: error_code:UnexpectedError reason:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed" code:2100 detail:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed""] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:337\ngithub.com/zilliztech/milvus-backup/cmd.init.func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:85\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:989\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:19\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.5/x64/src/runtime/proc.go:272"] workerpool: execute job restore_collection: restore data: restore_collection: restore partition data v1: restore_collection: restore L0 segment: restore_collection: restore L0 segment bulk insert via grpc: restore_collection: failed to bulk insert via grpc: client: bulk insert failed: client: operation failed: error_code:UnexpectedError reason:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed" code:2100 detail:"unexpect file type, files=[restore-temp-restore_2025_02_17_09_35_50_465377159-default-LLM_MANAGER_LARGE_20250217/large_backup20250217/binlogs/delta_log/451680931584957868/451680931682450528/451680931682450533/]: importing data failed"
And restore this backup by milvus-backup 0.4.13, encounter this problem:
[2025/02/18 01:46:42.918 +00:00] [ERROR] [core/backup_impl_restore_backup.go:368] ["executeRestoreCollectionTask failed"] [TargetDBName=default] [TargetCollectionName=LLM_MANAGER_LARGE_20250218] [error="no binlog to import, input=[paths:"restore-temp-f72f5203-ed97-11ef-8ff7-0242ac120005-default-LLM_MANAGER_LARGE_20250218/backup/large_backup20250217/binlogs/insert_log/451680931584957868/451680931682450528/451680931682450533/" paths:"" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:"restore-temp-f72f5203-ed97-11ef-8ff7-0242ac120005-default-LLM_MANAGER_LARGE_20250218/backup/large_backup20250217/binlogs/insert_log/451680931584957868/451680931682450528/451680931682450533/" paths:"" ]: invalid parameter\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/[email protected]/client/collection.go:39\n | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n | \t/home/runner/go/pkg/mod/github.com/wayblink/milvus-sdk-go/[email protected]/client/insert.go:428\n | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:122\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:724\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func4\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:657\n | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func5\n | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:674\n | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:"restore-temp-f72f5203-ed97-11ef-8ff7-0242ac120005-default-LLM_MANAGER_LARGE_20250218/backup/large_backup20250217/binlogs/insert_log/451680931584957868/451680931682450528/451680931682450533/" paths:"" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:368\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"] [2025/02/18 01:46:42.918 +00:00] [ERROR] [core/backup_impl_restore_backup.go:326] ["execute restore collection fail"] [backupId=b4b3cabf-ed11-11ef-9315-0242ac12000d] [error="workerpool: execute job no binlog to import, input=[paths:"restore-temp-f72f5203-ed97-11ef-8ff7-0242ac120005-default-LLM_MANAGER_LARGE_20250218/backup/large_backup20250217/binlogs/insert_log/451680931584957868/451680931682450528/451680931682450533/" paths:"" ]: invalid parameter"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:326\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:83\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"] workerpool: execute job no binlog to import, input=[paths:"restore-temp-f72f5203-ed97-11ef-8ff7-0242ac120005-default-LLM_MANAGER_LARGE_20250218/backup/large_backup20250217/binlogs/insert_log/451680931584957868/451680931682450528/451680931682450533/" paths:"" ]: invalid parameter
The new version still has the "channel not found error", only when using -f can create backup.
[2025/02/18 02:13:45.744 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2025/02/18 02:13:45.744 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2025/02/18 02:13:45.746 +00:00] [INFO] [core/backup_impl_create_backup.go:44] ["receive CreateBackupRequest"] [requestId=fb889c17-ed9d-11ef-9cf3-0242ac12000d] [backupName=large_backup_preprod20250218] [collections="[LLM_MANAGER_LARGE_PREPROD01]"] [databaseCollections=] [async=false] [force=false] [metaOnly=false] [2025/02/18 02:13:45.746 +00:00] [INFO] [core/backup_context.go:87] ["{Base:0xc00044ec08 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"] [2025/02/18 02:13:45.746 +00:00] [INFO] [core/backup_context.go:88] ["{Base:0xc00044ec08 Enabled:true DebugMode:false SimpleResponse:true}"] [2025/02/18 02:13:45.757 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=a-bucket] [root=backup] [2025/02/18 02:13:45.766 +00:00] [INFO] [client/grpc.go:219] ["New milvus grpc client"] [host=172.17.132.197:31530] [2025/02/18 02:13:45.766 +00:00] [WARN] [grpclog/grpclog.go:155] ["[core]Adjusting keepalive ping interval to minimum period of 10s"] [2025/02/18 02:13:45.773 +00:00] [WARN] [core/backup_impl_create_backup.go:616] ["Pause Milvus GC Error:This warn won't fail the backup process. Pause GC can protect data not to be GCed during backup, it is necessary to backup very large data(cost more than a hour)."] [error="Get "http://localhost:9091/management/datacoord/garbage_collection/pause?pause_seconds=7200": dial tcp 127.0.0.1:9091: connect: connection refused"] [2025/02/18 02:13:45.777 +00:00] [INFO] [core/backup_impl_create_backup.go:682] ["collections to backup"] [collections="[default.LLM_MANAGER_LARGE_PREPROD01]"] [2025/02/18 02:13:45.777 +00:00] [INFO] [core/backup_impl_create_backup.go:243] ["start backup collection"] [db=default] [collection=LLM_MANAGER_LARGE_PREPROD01] [2025/02/18 02:13:45.783 +00:00] [INFO] [core/backup_impl_create_backup.go:303] ["try to get index"] [collection_name=LLM_MANAGER_LARGE_PREPROD01] [2025/02/18 02:13:45.791 +00:00] [INFO] [core/backup_impl_create_backup.go:310] ["List index"] [collection_name=LLM_MANAGER_LARGE_PREPROD01] [indexes="[{"index_name":"default","indexID":451680931590267379,"params":[{"key":"index_type","value":"IVF_FLAT"},{"key":"metric_type","value":"IP"},{"key":"params","value":"{\"nlist\":\"128\"}"}],"field_name":"embedding","indexed_rows":2948298,"total_rows":2948298,"state":3}]"] [2025/02/18 02:13:45.848 +00:00] [INFO] [core/backup_impl_create_backup.go:395] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=default] [collectionName=LLM_MANAGER_LARGE_PREPROD01] [segmentNumBeforeFlush=979] [2025/02/18 02:16:37.589 +00:00] [ERROR] [core/backup_impl_create_backup.go:401] ["fail to flush the collection"] [databaseName=default] [collectionName=LLM_MANAGER_LARGE_PREPROD01] [error="client: flush failed: client: operation failed: error_code:UnexpectedError reason:"channel not found[channel=by-dev-rootcoord-dml_6_451680931584958393v0]" code:500 detail:"channel not found[channel=by-dev-rootcoord-dml_6_451680931584958393v0]""] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).backupCollectionPrepare\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:401\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func1.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:690\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeCreateBackup.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:706\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:71\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"] [2025/02/18 02:16:37.591 +00:00] [WARN] [core/backup_impl_create_backup.go:634] ["Resume Milvus GC Error:This warn won't fail the backup process. Pause GC can protect data not to be GCed during backup, it is necessary to backup very large data(cost more than a hour)."] [error="Get "http://localhost:9091/management/datacoord/garbage_collection/resume": dial tcp 127.0.0.1:9091: connect: connection refused"] client: flush failed: client: operation failed: error_code:UnexpectedError reason:"channel not found[channel=by-dev-rootcoord-dml_6_451680931584958393v0]" code:500 detail:"channel not found[channel=by-dev-rootcoord-dml_6_451680931584958393v0]"
Does the latest version support milvus 2.4.1 and 2.2.6? Previously these is a version compatible talbe, but now it is disappeared.
You can use Milvus Backup 0.5.1 to back up Milvus 2.4.1 and 2.2.6. However, it is recommended to restore the backup to the latest version of Milvus to ensure compatibility and optimal performance.
Regarding the missing compatibility table, we will check and update the relevant documentation. Thanks for your feedback!
The new version still has the "channel not found error", only when using -f can create backup.
It looks like there might be an issue with Milvus. If you don’t have continuous writes, you can still use the -f flag to perform the backup.