matrixone
matrixone copied to clipboard
matrixorigin
[Bug]: mo-service panic in docker env
Is there an existing issue for the same bug?
- [X] I have checked the existing issues.
Environment
- Version or commit-id: https://github.com/matrixorigin/matrixone/commit/769795c979f965480b904c10e4c87c3fb7168e8e
- Host: Macbook Pro, 2.3 GHz 8-Core Intel Core i9, 16 GB 2667 MHz DDR4
- Base Image: ubuntu
Actual Behavior
mo-service panic with the following log:
docker run --network host -v `pwd`:`pwd` -w `pwd` -it aylei/mo-service -cfg etc/bootstrap-example/log-node-1.toml
{"level":"DEBUG","time":"2022/08/08 03:10:13.509193 +0000","caller":"logutil/internal.go:34","msg":"MO logger init, level=debug, log file="}
hakeeper client cfg: { [127.0.0.1:32001 127.0.0.1:32011 127.0.0.1:32021]}2022-08-08 03:10:13.509306 I | dragonboat: go version: go1.18.5, linux/amd64
2022-08-08 03:10:13.509333 I | dragonboat: dragonboat version: 4.0.0 (Dev)
2022-08-08 03:10:13.509339 W | config: mutual TLS disabled, communication is insecure
2022-08-08 03:10:13.509376 I | config: using default EngineConfig
2022-08-08 03:10:13.509385 I | config: using default LogDBConfig
2022-08-08 03:10:13.509415 I | dragonboat: DeploymentID set to 1
2022-08-08 03:10:13.579576 I | dragonboat: LogDB info received, shard 0, busy false
2022-08-08 03:10:13.600932 I | dragonboat: LogDB info received, shard 1, busy false
2022-08-08 03:10:13.623093 I | dragonboat: LogDB info received, shard 2, busy false
2022-08-08 03:10:13.644010 I | dragonboat: LogDB info received, shard 3, busy false
2022-08-08 03:10:13.663486 I | dragonboat: LogDB info received, shard 4, busy false
2022-08-08 03:10:13.684757 I | dragonboat: LogDB info received, shard 5, busy false
2022-08-08 03:10:13.703537 I | dragonboat: LogDB info received, shard 6, busy false
2022-08-08 03:10:13.722827 I | dragonboat: LogDB info received, shard 7, busy false
2022-08-08 03:10:13.766531 I | dragonboat: LogDB info received, shard 8, busy false
2022-08-08 03:10:13.793118 I | dragonboat: LogDB info received, shard 9, busy false
2022-08-08 03:10:13.814569 I | dragonboat: LogDB info received, shard 10, busy false
2022-08-08 03:10:13.834193 I | dragonboat: LogDB info received, shard 11, busy false
2022-08-08 03:10:13.852057 I | dragonboat: LogDB info received, shard 12, busy false
2022-08-08 03:10:13.870202 I | dragonboat: LogDB info received, shard 13, busy false
2022-08-08 03:10:13.888689 I | dragonboat: LogDB info received, shard 14, busy false
2022-08-08 03:10:13.906875 I | logdb: using plain logdb
2022-08-08 03:10:13.906983 I | dragonboat: LogDB info received, shard 15, busy false
2022-08-08 03:10:13.912236 I | dragonboat: logdb memory limit: 8192 MBytes
2022-08-08 03:10:13.915750 I | dragonboat: NodeHost ID: 7c4dccb4-4d3c-41f8-b482-5251dc7a41bf
2022-08-08 03:10:13.915775 I | dragonboat: AddressByNodeHostID: true, use gossip based node registry
2022-08-08 03:10:13.915792 I | registry: gossip probe interval set to 50ms
2022-08-08 03:10:13.915826 I | registry: gossip bind address 127.0.0.1 port 32002
2022-08-08 03:10:13.915832 I | registry: gossip advertise address 127.0.0.1 port 32002
2022-08-08 03:10:13.917236 I | dragonboat: filesystem error injection mode enabled: false
2022-08-08 03:10:13.917545 I | transport: transport type: go-tcp-transport
2022-08-08 03:10:13.917624 I | dragonboat: transport type: go-tcp-transport
2022-08-08 03:10:13.917632 I | dragonboat: logdb type: Tee
2022-08-08 03:10:13.917637 I | dragonboat: nodehost address: 127.0.0.1:32000
2022-08-08 03:10:13.917644 I | LogService: HAKeeper LogStoreTimeout: 20s, DnStoreTimeout: 10s
2022-08-08 03:10:13.918122 I | LogService: logservice truncation worker started
2022-08-08 03:10:13.918606 I | LogService: logservice heartbeat worker started
bootstrapping hakeeper...
2022-08-08 03:10:13.933094 I | tan: /Users/alei/go/src/github.com/matrixorigin/matrixone/node-1-data/docker-desktop/00000000000000000001/tee-tan/tandb/node-0-131072 creating a new tan db
2022-08-08 03:10:13.941285 I | tan: /Users/alei/go/src/github.com/matrixorigin/matrixone/node-1-data/docker-desktop/00000000000000000001/tee-tan/tandb/node-0-131072 on disk files [CURRENT MANIFEST-000001]
2022-08-08 03:10:13.941321 I | tan: /Users/alei/go/src/github.com/matrixorigin/matrixone/node-1-data/docker-desktop/00000000000000000001/tee-tan/tandb/node-0-131072 logFiles to rebuild: map[]
2022-08-08 03:10:13.941327 I | tan: /Users/alei/go/src/github.com/matrixorigin/matrixone/node-1-data/docker-desktop/00000000000000000001/tee-tan/tandb/node-0-131072 indexFiles: []
2022-08-08 03:10:13.943327 C | LogDB: conflict errors, [00000:31072], e1 operation not supported, e2 <nil>
panic: conflict errors, [00000:31072], e1 operation not supported, e2 <nil>
goroutine 422 [running]:
github.com/lni/goutils/logutil/capnslog.(*PackageLogger).Panicf(0xd?, {0x1404b4e?, 0xc000010438?}, {0xc0044e63f0?, 0xc00003dc40?, 0xc0045f8638?})
/go/pkg/mod/github.com/lni/[email protected]/logutil/capnslog/pkg_logger.go:88 +0xbb
github.com/lni/dragonboat/v4/logger.(*capnsLog).Panicf(0xc00003dc20?, {0x1404b4e?, 0x417567?}, {0xc0044e63f0?, 0x1214320?, 0xc00038c701?})
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/logger/capnslogger.go:74 +0x26
github.com/lni/dragonboat/v4/logger.(*dragonboatLogger).Panicf(0xc00038c720?, {0x1404b4e, 0x21}, {0xc0044e63f0, 0x3, 0x3})
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/logger/logger.go:132 +0x57
github.com/lni/dragonboat/v4/internal/logdb/tee.assertSameError(0x0?, 0x0?, {0x159ebc0?, 0x1d52f78}, {0x0?, 0x0})
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/internal/logdb/tee/tee.go:47 +0x1c4
github.com/lni/dragonboat/v4/internal/logdb/tee.(*LogDB).GetSnapshot(0xc0003a61e0, 0xc0045f8c68?, 0x46bc7b?)
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/internal/logdb/tee/tee.go:280 +0x228
github.com/lni/dragonboat/v4.(*snapshotter).GetSnapshotFromLogDB(0x11?)
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/snapshotter.go:189 +0x8c
github.com/lni/dragonboat/v4.(*snapshotter).processOrphans(0xc0000a9090)
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/snapshotter.go:274 +0x8b
github.com/lni/dragonboat/v4.(*NodeHost).startShard(0xc00032ce00, 0xc000353d10, 0x0, 0xc000100400?, {0x20000, 0x0, 0x1, 0x1, 0xa, 0x1, ...}, ...)
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/nodehost.go:1575 +0x8da
github.com/lni/dragonboat/v4.(*NodeHost).StartReplica(0xc0001aa078?, 0x24?, 0x2b?, 0x6?, {0x20000, 0x0, 0x1, 0x1, 0xa, 0x1, ...})
/go/pkg/mod/github.com/lni/dragonboat/[email protected]/nodehost.go:506 +0xe5
github.com/matrixorigin/matrixone/pkg/logservice.(*store).startHAKeeperReplica(0xc00038a400, 0x20000, 0xc000221110?, 0xb0?)
/go/src/github.com/matrixorigin/matrixone/pkg/logservice/store.go:193 +0xc7
github.com/matrixorigin/matrixone/pkg/logservice.(*Service).BootstrapHAKeeper(_, {_, _}, {{0x0, 0x0}, 0x1, {0xc0000be990, 0x24}, 0x0, {0xc000045e90, ...}, ...})
/go/src/github.com/matrixorigin/matrixone/pkg/logservice/service_bootstrap.go:33 +0x8a
main.startLogService.func1({0x15aad98, 0xc000287a40})
/go/src/github.com/matrixorigin/matrixone/cmd/mo-service/main.go:113 +0x110
github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
/go/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:252 +0x79
created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask
/go/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:247 +0x12c
Expected Behavior
mo-service runs normally
Steps to Reproduce
No response
Additional information
The darwin/amd64 binary built by the same code runs normally in my macbook without docker
@aylei could u pls test it at the latest commit and update the issue status, thx
