matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[Bug]: mo-server crash when removing segment files

Open LeftHandCold opened this issue 3 years ago • 0 comments

Is there an existing issue for the same bug?

  • [X] I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):e1e7d5c7e54b516e58d23a5c90ea885576525ca6
- Hardware parameters:
- OS type:centos macos
- Others:

Actual Behavior

The path of the segment is normal when it is created. After SoftDelete, the path of the segment is abnormal at the time of GC, causing the remove to fail.

Expected Behavior

No response

Steps to Reproduce

Execute bvt repeatedly.
rarely recurs.

Additional information

{"level":"INFO","time":"2022/06/28 21:37:15.125114 +0800","caller":"db/gcop.go:39","msg":"[GCBLK] | [Appendable]BLOCK[<0:23419-20740-20790-0-0>] | Removed"} {"level":"INFO","time":"2022/06/28 21:37:15.125121 +0800","caller":"segmentio/segment.go:275","msg":"Destroying Driver 20740"} {"level":"INFO","time":"2022/06/28 21:37:15.125138 +0800","caller":"segmentio/driver.go:163","msg":"Unmount Driver: z\ufffd\ufffd\u0003\ufffd\ufffd\ufffd?e/20740.seg"} {"level":"DEBUG","time":"2022/06/28 21:37:15.125165 +0800","caller":"segmentio/driver.go:307","msg":" z\ufffd\ufffd\u0003\ufffd\ufffd\ufffd?e/20740.seg-0xc00020df00 | SegmentFile | Destroying | Null-1-11 | Log Level1 ffffffffffffffc0-ffffffffffffffff"} {"level":"INFO","time":"2022/06/28 21:37:15.125195 +0800","caller":"db/gcop.go:67","msg":"[GCSEG] | [Appendable]SEGMENT[<0:23419-20740-0-0-0>] | BLKS=[<0:23419-20740-20790-0-0>,] | Removed"} {"level":"INFO","time":"2022/06/28 21:37:15.125209 +0800","caller":"db/gcop.go:95","msg":"[GCTABLE] | TABLE[Op=SoftDelete][ID=23419][265681=>265693]<Index[86173:0/1]>[name=t1] | Ended: | SEGS=[<0:23419-20740-0-0-0>,]"} {"level":"INFO","time":"2022/06/28 21:37:15.125225 +0800","caller":"db/gcop.go:118","msg":"[GCDB] | DB[Op=SoftDelete][ID=3108][265486=>265717]<Index[86182:0/1]>[name=smallint] | Ended: | TABLES=[<0:23421-0-0-0-0>,<0:23420-0-0-0-0>,<0:23419-0-0-0-0>,]"} panic: remove zì³^C<8a><8c>Ý?e/20740.seg: no such file or directory

goroutine 228 [running]: github.com/matrixorigin/matrixone/pkg/vm/engine/tae/dataio/segmentio.(*Driver).Destroy(0xc00020dea0) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/dataio/segmentio/driver.go:176 +0x1b4 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/dataio/segmentio.(*segmentFile).Destroy(0xc013604420) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/dataio/segmentio/segment.go:285 +0x18f github.com/matrixorigin/matrixone/pkg/vm/engine/tae/dataio/segmentio.(*segmentFile).close(0xc0040b7800?) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/dataio/segmentio/segment.go:272 +0x19 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/common.(*RefHelper).Unref(0xc0040b7800?) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/common/refs.go:50 +0x36 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables.(*dataSegment).Destory(0xc013955ef0?) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/tables/segment.go:60 +0x4f github.com/matrixorigin/matrixone/pkg/vm/engine/tae/catalog.(*SegmentEntry).DestroyData(...) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/catalog/segment.go:388 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.gcSegmentClosure.func1() /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/db/gcop.go:81 +0x2bc github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.gcTableClosure.func1() /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/db/gcop.go:102 +0x23c github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.gcDatabaseClosure.func1() /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/db/gcop.go:124 +0x1f0 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks.(*FnTask).Execute(0xc000280180?) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/tasks/types.go:122 +0x1b github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/ops.(*Op).OnExec(0xc005670b60) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/tasks/ops/ops.go:90 +0x6f github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(*OpWorker).onOp(0xc0003df6c0, {0x7fe0d48ed998, 0xc0174005a0}) /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/tasks/worker/worker.go:214 +0x30 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(*OpWorker).Start.func1() /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/tasks/worker/worker.go:144 +0xad created by github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(*OpWorker).Start /mnt/vdb/sjw/tae/matrixone/pkg/vm/engine/tae/tasks/worker/worker.go:136 +0xb6

image image image

LeftHandCold avatar Jun 29 '22 02:06 LeftHandCold