milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: Crash with SIGABRT when close writer
Is there an existing issue for this?
- [x] I have searched the existing issues
Environment
- Milvus version: 6694fe2f10397964d3191960cb2b25d0dd3ad9ac
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
2025-11-12 00:23:33.319 /workspace/source/cmake_build/thirdparty/milvus-storage/milvus-storage-src/cpp/include/milvus-storage/filesystem/io/io_util.h:38: IOError: When destroying file of type N14milvus_storage18CustomOutputStreamE: When completing multiple part upload for key 'file/insert_log/462127726720871115/462127726720871116/462127726719867942/0/462127726723114412' in bucket 'milvus-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2025-11-12 00:23:33.320 terminate called after throwing an instance of 'std::runtime_error'
2025-11-12 00:23:33.320 what(): When destroying file of type N14milvus_storage18CustomOutputStreamE: When completing multiple part upload for key 'file/insert_log/462127726720871115/462127726720871116/462127726719867942/0/462127726723114412' in bucket 'milvus-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2025-11-12 00:23:33.320 /workspace/source/cmake_build/thirdparty/milvus-storage/milvus-storage-src/cpp/include/milvus-storage/filesystem/io/io_util.h:38: IOError: When destroying file of type N14milvus_storage18CustomOutputStreamE: When completing multiple part upload for key 'file/insert_log/462127726720871139/462127726720871140/462127726719867943/0/462127726723114434' in bucket 'milvus-bucket': AWS Error NO_SUCH_UPLOAD during CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.
2025-11-12 00:23:33.320 terminate called recursively
2025-11-12 00:23:33.322 SIGABRT: abort
2025-11-12 00:23:33.322 PC=0x7fa8e3def9fc m=356 sigcode=18446744073709551610
2025-11-12 00:23:33.322 signal arrived during cgo execution
2025-11-12 00:23:33.322
2025-11-12 00:23:33.322 goroutine 440680 gp=0x10c008fdf6c0 m=356 mp=0x10c00628d008 [syscall]:
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e3def9fc
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e3d9b475
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e3d817f2
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e412bb9d
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e413720b
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e41361e8
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e4136958
2025-11-12 00:23:34.921 non-Go function
2025-11-12 00:23:34.921 pc=0x7fa8e3f98883
2025-11-12 00:23:34.921 runtime.cgocall(0x68b0bd0, 0x10c00411d6f0)
2025-11-12 00:23:34.921 /usr/local/go/src/runtime/cgocall.go:167 +0x4b fp=0x10c00411d6c8 sp=0x10c00411d690 pc=0x286694b
2025-11-12 00:23:34.921 github.com/milvus-io/milvus/internal/storagev2/packed._Cfunc_CloseWriter(0x51100053ccc0)
2025-11-12 00:23:34.921 _cgo_gotypes.go:226 +0x4b fp=0x10c00411d6f0 sp=0x10c00411d6c8 pc=0x4ef608b
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/storagev2/packed.(*PackedWriter).Close.func1(...)
2025-11-12 00:23:34.922 /workspace/source/internal/storagev2/packed/packed_writer.go:154
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/storagev2/packed.(*PackedWriter).Close(0x1?)
2025-11-12 00:23:34.922 /workspace/source/internal/storagev2/packed/packed_writer.go:154 +0x35 fp=0x10c00411d738 sp=0x10c00411d6f0 pc=0x4efbd55
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/storage.(*packedRecordWriter).Close(0x10c0038da980)
2025-11-12 00:23:34.922 /workspace/source/internal/storage/serde_events_v2.go:188 +0x39 fp=0x10c00411d7f0 sp=0x10c00411d738 pc=0x5269a99
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/flushcommon/syncmgr.(*BulkPackWriterV2).writeInserts(0x10c00411dd80, {0x84b01e0?, 0xbb13b60?}, 0x10c003c51c80)
2025-11-12 00:23:34.922 /workspace/source/internal/flushcommon/syncmgr/pack_writer_v2.go:185 +0x619 fp=0x10c00411da10 sp=0x10c00411d7f0 pc=0x58f5bb9
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/flushcommon/syncmgr.(*BulkPackWriterV2).Write(0x10c00411dd80, {0x84b01e0, 0xbb13b60}, 0x10c003c51c80)
2025-11-12 00:23:34.922 /workspace/source/internal/flushcommon/syncmgr/pack_writer_v2.go:88 +0x1b3 fp=0x10c00411db40 sp=0x10c00411da10 pc=0x58f4ed3
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/flushcommon/syncmgr.(*SyncTask).Run(0x10c0184352c0, {0x84b01e0, 0xbb13b60})
2025-11-12 00:23:34.922 /workspace/source/internal/flushcommon/syncmgr/task.go:143 +0x35f fp=0x10c00411dea0 sp=0x10c00411db40 pc=0x58f99bf
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/internal/flushcommon/syncmgr.(*keyLockDispatcher[...]).Submit.func1()
2025-11-12 00:23:34.922 /workspace/source/internal/flushcommon/syncmgr/key_lock_dispatcher.go:39 +0x88 fp=0x10c00411df20 sp=0x10c00411dea0 pc=0x58fc748
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/pkg/v2/util/conc.(*Pool[...]).Submit.func1()
2025-11-12 00:23:34.922 /workspace/source/pkg/util/conc/pool.go:82 +0x9e fp=0x10c00411df80 sp=0x10c00411df20 pc=0x58c901e
2025-11-12 00:23:34.922 github.com/panjf2000/ants/v2.(*goWorker).run.func1()
2025-11-12 00:23:34.922 /go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:73 +0x8a fp=0x10c00411dfe0 sp=0x10c00411df80 pc=0x51ab30a
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c00411dfe8 sp=0x10c00411dfe0 pc=0x2873a41
2025-11-12 00:23:34.922 created by github.com/panjf2000/ants/v2.(*goWorker).run in goroutine 67991
2025-11-12 00:23:34.922 /go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:50 +0x65
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 1 gp=0x10c000002380 m=nil [chan receive, 13 minutes]:
2025-11-12 00:23:34.922 runtime.gopark(0x277744813d097ac?, 0x7fa6701e6a28?, 0x8?, 0xf1?, 0x18?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:435 +0xce fp=0x10c00592f5d0 sp=0x10c00592f5b0 pc=0x286adce
2025-11-12 00:23:34.922 runtime.chanrecv(0x10c00183e5b0, 0x0, 0x1)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/chan.go:664 +0x445 fp=0x10c00592f648 sp=0x10c00592f5d0 pc=0x27fe025
2025-11-12 00:23:34.922 runtime.chanrecv1(0xbab0d50?, 0x7b0f1a0?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/chan.go:506 +0x12 fp=0x10c00592f670 sp=0x10c00592f648 pc=0x27fdbb2
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0x10c0018519f0)
2025-11-12 00:23:34.922 /workspace/source/cmd/roles/roles.go:554 +0x15a5 fp=0x10c00592fba8 sp=0x10c00592f670 pc=0x682b0a5
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xbb13b60, {0x10c0001a2960, 0x3, 0x3}, 0x10c00163ea10)
2025-11-12 00:23:34.922 /workspace/source/cmd/milvus/run.go:48 +0x2ae fp=0x10c00592fc68 sp=0x10c00592fba8 pc=0x683800e
2025-11-12 00:23:34.922 github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0x10c0001a2960, 0x3, 0x3})
2025-11-12 00:23:34.922 /workspace/source/cmd/milvus/milvus.go:60 +0x1e2 fp=0x10c00592fcd8 sp=0x10c00592fc68 pc=0x6837cc2
2025-11-12 00:23:34.922 main.main()
2025-11-12 00:23:34.922 /workspace/source/cmd/main.go:102 +0x285 fp=0x10c00592ff50 sp=0x10c00592fcd8 pc=0x683c405
2025-11-12 00:23:34.922 runtime.main()
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:283 +0x28b fp=0x10c00592ffe0 sp=0x10c00592ff50 pc=0x283358b
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c00592ffe8 sp=0x10c00592ffe0 pc=0x2873a41
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 17 gp=0x10c0000028c0 m=1 mp=0x10c000150008 [syscall, locked to thread]:
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c00015efe8 sp=0x10c00015efe0 pc=0x2873a41
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 2 gp=0x10c000002e00 m=nil [force gc (idle), 13 minutes]:
2025-11-12 00:23:34.922 runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:435 +0xce fp=0x10c00014afa8 sp=0x10c00014af88 pc=0x286adce
2025-11-12 00:23:34.922 runtime.goparkunlock(...)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:441
2025-11-12 00:23:34.922 runtime.forcegchelper()
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:348 +0xb3 fp=0x10c00014afe0 sp=0x10c00014afa8 pc=0x28338d3
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c00014afe8 sp=0x10c00014afe0 pc=0x2873a41
2025-11-12 00:23:34.922 created by runtime.init.7 in goroutine 1
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:336 +0x1a
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 18 gp=0x10c000182380 m=nil [GC sweep wait]:
2025-11-12 00:23:34.922 runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:435 +0xce fp=0x10c000146780 sp=0x10c000146760 pc=0x286adce
2025-11-12 00:23:34.922 runtime.goparkunlock(...)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:441
2025-11-12 00:23:34.922 runtime.bgsweep(0x10c000190000)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgcsweep.go:316 +0xdf fp=0x10c0001467c8 sp=0x10c000146780 pc=0x2819bbf
2025-11-12 00:23:34.922 runtime.gcenable.gowrap1()
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0x10c0001467e0 sp=0x10c0001467c8 pc=0x280dda5
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c0001467e8 sp=0x10c0001467e0 pc=0x2873a41
2025-11-12 00:23:34.922 created by runtime.gcenable in goroutine 1
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:204 +0x66
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 19 gp=0x10c000182540 m=nil [GC scavenge wait]:
2025-11-12 00:23:34.922 runtime.gopark(0xa4dc62?, 0x16ee197?, 0x0?, 0x0?, 0x0?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:435 +0xce fp=0x10c000146f78 sp=0x10c000146f58 pc=0x286adce
2025-11-12 00:23:34.922 runtime.goparkunlock(...)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:441
2025-11-12 00:23:34.922 runtime.(*scavengerState).park(0xbab2a40)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0x10c000146fa8 sp=0x10c000146f78 pc=0x2817469
2025-11-12 00:23:34.922 runtime.bgscavenge(0x10c000190000)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0x10c000146fc8 sp=0x10c000146fa8 pc=0x2817a19
2025-11-12 00:23:34.922 runtime.gcenable.gowrap2()
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:205 +0x25 fp=0x10c000146fe0 sp=0x10c000146fc8 pc=0x280dd45
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c000146fe8 sp=0x10c000146fe0 pc=0x2873a41
2025-11-12 00:23:34.922 created by runtime.gcenable in goroutine 1
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:205 +0xa5
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 20 gp=0x10c000182a80 m=nil [finalizer wait]:
2025-11-12 00:23:34.922 runtime.gopark(0x0?, 0x7e320f8?, 0x20?, 0x40?, 0x1000000010?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:435 +0xce fp=0x10c0020f1e30 sp=0x10c0020f1e10 pc=0x286adce
2025-11-12 00:23:34.922 runtime.runfinq()
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mfinal.go:196 +0x107 fp=0x10c0020f1fe0 sp=0x10c0020f1e30 pc=0x280cce7
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c0020f1fe8 sp=0x10c0020f1fe0 pc=0x2873a41
2025-11-12 00:23:34.922 created by runtime.createfing in goroutine 1
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mfinal.go:166 +0x3d
2025-11-12 00:23:34.922
2025-11-12 00:23:34.922 goroutine 21 gp=0x10c000182c40 m=nil [GC worker (idle), 13 minutes]:
2025-11-12 00:23:34.922 runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/proc.go:435 +0xce fp=0x10c000147738 sp=0x10c000147718 pc=0x286adce
2025-11-12 00:23:34.922 runtime.gcBgMarkWorker(0x10c000198150)
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:1423 +0xe9 fp=0x10c0001477c8 sp=0x10c000147738 pc=0x2810209
2025-11-12 00:23:34.922 runtime.gcBgMarkStartWorkers.gowrap1()
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:1339 +0x25 fp=0x10c0001477e0 sp=0x10c0001477c8 pc=0x28100e5
2025-11-12 00:23:34.922 runtime.goexit({})
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0x10c0001477e8 sp=0x10c0001477e0 pc=0x2873a41
2025-11-12 00:23:34.922 created by runtime.gcBgMarkStartWorkers in goroutine 1
2025-11-12 00:23:34.922 /usr/local/go/src/runtime/mgc.go:1339 +0x105
Expected Behavior
No response
Steps To Reproduce
Milvus Log
log No response
Anything else?
No response
@chyezh could you please provide a core dump file? Or how to reproduce?
