TDengine
TDengine copied to clipboard
Published
20 hours ago •
taosdata
taosdata
⚠️ Invalid Vgroup ID !
Bug Description
09/02 14:45:55.968649 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:201, term:1, msg-index:201, weak:0, code:0, state:102 leader, type:submit
09/02 14:45:55.968696 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:202, term:1, msg-index:202, weak:0, code:0, state:102 leader, type:submit
09/02 14:45:55.968775 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:204, term:2, msg-index:204, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 14:45:55.968849 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:206, term:3, msg-index:206, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 14:45:55.968914 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:208, term:4, msg-index:208, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 14:45:55.968982 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:210, term:5, msg-index:210, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 14:45:55.969044 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:212, term:6, msg-index:212, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 14:45:55.969099 00000046 VND vgId:17, commit-cb is excuted, fsm:0x7f17c012d410, index:214, term:7, msg-index:214, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 14:45:55.969147 00000046 DND TDengine initialized successfully
09/02 14:45:56.053993 00000048 DND ERROR vgId:16, msg:0x7f1794002030 failed to put into vnode queue since Invalid Vgroup ID, type:vnode-drop-ttl-stb qtype:3, gtid:0x0:0x3dc02ef2f0d50003
09/02 14:45:56.949248 00000063 MND dnode:1, mnode syncstate from error to leader
09/02 14:45:56.949274 00000063 MND dnode:1, from offline to online, memory avail:3451140916 total:3853242368 cores:2.00
09/02 14:45:56.954389 00000069 DND ERROR msg:0x7f1794003d50, failed to process since Node not deployed, gtid:0x0:0x3dc02ef2f45a0006
09/02 14:45:56.959275 00000069 VND vgId:17, start to commit, commit ID:1 version:216
09/02 14:45:56.963411 00000069 VND vgId:17, vnode info is saved, fname:/var/lib/taos/vnode/vnode17/vnode_tmp.json
taosd: /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c:1914: tsdbCmprData: Assertion `szIn > 0 && ppOut' failed.
Aborted (core dumped)
root@VM-16-2-debian:~# docker exec -it taos taos -pNoNews14GoodNews
Error response from daemon: Container 6e5a3e3c06aa0fee7a121c438a5bd5e1a634a232a23d5a76a4c7dc548af2b252 is restarting, wait until the container is running
To Reproduce
occur suddenly? i don't known how it happend!
Expected Behavior
???
Environment (please complete the following information):
- OS: [docker@debian]
- Memory, CPU, current Disk Space
root@VM-16-2-debian:~# free -h
total used free shared buff/cache available
Mem: 3.6Gi 2.4Gi 351Mi 68Mi 908Mi 949Mi
Swap: 0B 0B 0B
root@VM-16-2-debian:~# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 1.8G 24K 1.8G 1% /dev/shm
tmpfs 735M 1.5M 734M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda1 79G 14G 62G 19% /
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/2e699df2db041936caaba22d5756f6b70839dc88cc878db26589a82720ab4f25/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/9f6b7a9b081f98663b766f548152849f8af4db6c06b7e0cb9521cca52b2fc34f/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/8c783467ef673a443470a425333dd8d29d21262abb8773eb655d4dbd64c8a52c/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/9147c74c52e965f2af6aa5bc7156607e43fa939db2b6f9e32938e26a27ff6169/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/a6d7e55f1917bf156e3d24b2208fdd2e0223b420061b634b3a4691d9c4e95e9f/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/8350702f10428bb71faf55c14efbe1937d85a402e3beb4eb01e84df8bc97b581/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/8c7549b2f8f127286cd1e5792fdee709c5df4165b79e754920934303f1fa8260/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/5326ca1939f229f821932438d25d793e70311e4ca90eae54c9038cd8c87e2213/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/22e7f5d4c008baeff3447c64e919b71f2c3605e58275679d8fcdebdef1a7f229/merged
tmpfs 368M 0 368M 0% /run/user/0
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/84591b62685ef27cd0ecb1d2ae95b5bedd745a581974904ab477ea0c8aa28556/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/b3a35d239b94c77aee54a8ba59666d9ff51f4595970407e2d8eede68948af8f3/merged
overlay 79G 14G 62G 19% /var/lib/docker/overlay2/a9b8ab96f71ff3d95b54019e414a843c9524fd50081245bd825f53c395e7f0a6/merged
root@VM-16-2-debian:~#
- TDengine Version [
3.0.0.1]
The version of TDengine please ~
ah , 3.0.0.1
this is the compose
taos:
image: tdengine/tdengine:3.0.0.1
container_name: taos
hostname: taos
restart: always
ports:
- "${PORT_TAOS_TCP}:6030"
- "${PORT_TAOS_REST}:6041"
environment:
PASSWORD: ${PASSWORD}
volumes:
- /etc/localtime:/etc/localtime:ro
- ${DATA}/taos/etc:/etc/taos
- ${DATA}/taos/data:/var/lib/taos
- ${DATA}/taos/log:/var/log/taos
deploy:
placement:
constraints: [node.host == manager]
and the env PASSWORD is not work
May I know :
-
this is the first time you start up this image ?
-
did you execute any writing operation after start this image .
-
could you restart it again to normal state without coredump ?
May I know :
1. this is the first time you start up this image ?
yes
2. did you execute any writing operation after start this image .
yes, i both use emqx `rest api` and `cli sql` to test , maybe some invalid data error request before the server down
3. could you restart it again to normal state without coredump ?
i had a try to restart the container 3 to 4 times, but it not works, so i `docker rm -f taos && rm -rf /opt/data/taos` to destory and clean all the data , then repub again
then I guess this issue can be reproduced , could you run this in the physical machine ， it is easy to get core file
Then we need to use gdb to check stack information.
addtion
i reproduced
taos> drop synron_cloud_store;
DB error: syntax error near "synron_cloud_store;" (0.000045s)
taos> drop database synron_cloud_store;
DB error: Unable to establish connection (0.166872s)
taos> root@VM-16-2-debian:~# docker logs -f taos
...
09/02 16:44:57.113348 00000046 DND TDengine initialized successfully
09/02 16:44:58.082095 00000063 MND dnode:1, mnode syncstate from error to leader
09/02 16:44:58.082122 00000063 MND dnode:1, from offline to online, memory avail:3451140916 total:3853242368 cores:2.00
09/02 16:44:58.092841 00000069 VND vgId:2, start to commit, commit ID:1 version:41
09/02 16:44:58.096573 00000069 VND vgId:2, vnode info is saved, fname:/var/lib/taos/vnode/vnode2/vnode_tmp.json
taosd: /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c:1914: tsdbCmprData: Assertion `szIn > 0 && ppOut' failed.
Aborted (core dumped)
sysctl: setting key "kernel.core_pattern": Read-only file system
config file not found
09/02 16:44:59.965742 00000035 TAOS_ADAPTER info "start server:00000035" model=main
09/02 16:44:59.966099 00000035 TAOS_ADAPTER info "init plugin opentsdb_telnet/v1" model=plugin
09/02 16:44:59.967244 00000035 TAOS_ADAPTER info "opentsdb_telnet disabled" model=opentsdb_telnet
09/02 16:44:59.967262 00000035 TAOS_ADAPTER info "init plugin prometheus/v1" model=plugin
09/02 16:44:59.967435 00000035 TAOS_ADAPTER info "init plugin statsd/v1" model=plugin
09/02 16:44:59.969234 00000035 TAOS_ADAPTER info "init plugin collectd/v1" model=plugin
09/02 16:44:59.970404 00000035 TAOS_ADAPTER info "init plugin influxdb/v1" model=plugin
09/02 16:44:59.970586 00000035 TAOS_ADAPTER info "init plugin node_exporter/v1" model=plugin
09/02 16:44:59.972312 00000035 TAOS_ADAPTER info "node_exporter disabled" model=NodeExporter
09/02 16:44:59.972330 00000035 TAOS_ADAPTER info "init plugin opentsdb/v1" model=plugin
09/02 16:44:59.972514 00000035 TAOS_ADAPTER info "all plugin init finish" model=plugin
09/02 16:44:59.972959 00000035 TAOS_ADAPTER info "UDP listening on "[::]:6044"" model=statsd
09/02 16:44:59.973132 00000035 TAOS_ADAPTER info "Started the statsd service on ":6044"" model=statsd
09/02 16:44:59.973194 00000035 TAOS_ADAPTER info "all plugin start finish" model=plugin
09/02 16:44:59.974098 00000035 TAOS_ADAPTER info "server on : 6041" model=main
run TDengine with single node.
09/02 16:45:00.243084 00000046 UTL fail to load apoll url
09/02 16:45:00.243181 00000046 UTL failed to load from cfg file /etc/taos/taos.cfg since No such file or directory, use default parameters
09/02 16:45:00.243196 00000046 UTL failed to load env file: .env
09/02 16:45:00.243203 00000046 UTL load from env variables cfg success
09/02 16:45:00.243596 00000046 UTL fail to load apoll url
09/02 16:45:00.243614 00000046 UTL failed to load from cfg file /etc/taos/taos.cfg since No such file or directory, use default parameters
09/02 16:45:00.243621 00000046 UTL failed to load env file: .env
09/02 16:45:00.243627 00000046 UTL load from env variables cfg success
09/02 16:45:00.243724 00000046 UTL global config
09/02 16:45:00.243733 00000046 UTL =================================================================
09/02 16:45:00.243738 00000046 UTL default firstEp taos:6030
09/02 16:45:00.243741 00000046 UTL default secondEp taos:6030
09/02 16:45:00.243744 00000046 UTL default fqdn taos
09/02 16:45:00.243748 00000046 UTL default serverPort 6030
09/02 16:45:00.243752 00000046 UTL default tempDir /tmp
09/02 16:45:00.243755 00000046 UTL default minimalTmpDirGB 1.00
09/02 16:45:00.243762 00000046 UTL default shellActivityTimer 3
09/02 16:45:00.243766 00000046 UTL default compressMsgSize -1
09/02 16:45:00.243770 00000046 UTL default compressColData -1
09/02 16:45:00.243773 00000046 UTL default queryPolicy 1
09/02 16:45:00.243777 00000046 UTL default querySmaOptimize 0
09/02 16:45:00.243781 00000046 UTL default smlChildTableName
09/02 16:45:00.243785 00000046 UTL default smlTagName _tag_null
09/02 16:45:00.243788 00000046 UTL default smlDataFormat 1
09/02 16:45:00.243792 00000046 UTL default numOfTaskQueueThreads 4
09/02 16:45:00.243796 00000046 UTL default dataDir /var/lib/taos
09/02 16:45:00.243821 00000046 UTL default minimalDataDirGB 2.00
09/02 16:45:00.243826 00000046 UTL default supportVnodes 4
09/02 16:45:00.243830 00000046 UTL default maxShellConns 50000
09/02 16:45:00.243833 00000046 UTL default statusInterval 1
09/02 16:45:00.243837 00000046 UTL default minSlidingTime 10
09/02 16:45:00.243840 00000046 UTL default minIntervalTime 1
09/02 16:45:00.243844 00000046 UTL default maxNumOfDistinctRes 10000000
09/02 16:45:00.243848 00000046 UTL default countAlwaysReturnValue 1
09/02 16:45:00.243851 00000046 UTL default queryBufferSize -1
09/02 16:45:00.243855 00000046 UTL default printAuth 0
09/02 16:45:00.243858 00000046 UTL default multiProcess 0
09/02 16:45:00.243862 00000046 UTL default mnodeShmSize 20972544
09/02 16:45:00.243866 00000046 UTL default vnodeShmSize 104858624
09/02 16:45:00.243869 00000046 UTL default qnodeShmSize 41944064
09/02 16:45:00.243873 00000046 UTL default snodeShmSize 41944064
09/02 16:45:00.243876 00000046 UTL default bnodeShmSize 41944064
09/02 16:45:00.243879 00000046 UTL default numOfShmThreads 1
09/02 16:45:00.243882 00000046 UTL default numOfRpcThreads 1
09/02 16:45:00.243885 00000046 UTL default numOfCommitThreads 2
09/02 16:45:00.243889 00000046 UTL default numOfMnodeReadThreads 1
09/02 16:45:00.243892 00000046 UTL default numOfVnodeQueryThreads 4
09/02 16:45:00.243895 00000046 UTL default numOfVnodeStreamThreads 4
09/02 16:45:00.243899 00000046 UTL default numOfVnodeFetchThreads 4
09/02 16:45:00.243941 00000046 UTL default numOfVnodeWriteThreads 2
09/02 16:45:00.243958 00000046 UTL default numOfVnodeSyncThreads 16
09/02 16:45:00.243974 00000046 UTL default numOfQnodeQueryThreads 4
09/02 16:45:00.243995 00000046 UTL default numOfQnodeFetchThreads 4
09/02 16:45:00.244011 00000046 UTL default numOfSnodeSharedThreads 2
09/02 16:45:00.244049 00000046 UTL default numOfSnodeUniqueThreads 2
09/02 16:45:00.244068 00000046 UTL default rpcQueueMemoryAllowed 385324236
09/02 16:45:00.244090 00000046 UTL default monitor 1
09/02 16:45:00.244112 00000046 UTL default monitorInterval 30
09/02 16:45:00.244128 00000046 UTL default monitorFqdn
09/02 16:45:00.244143 00000046 UTL default monitorPort 6043
09/02 16:45:00.244168 00000046 UTL default monitorMaxLogs 100
09/02 16:45:00.244184 00000046 UTL default monitorComp 0
09/02 16:45:00.244200 00000046 UTL default telemetryReporting 1
09/02 16:45:00.244216 00000046 UTL default telemetryInterval 86400
09/02 16:45:00.244249 00000046 UTL default telemetryServer telemetry.taosdata.com
09/02 16:45:00.244268 00000046 UTL default telemetryPort 80
09/02 16:45:00.244284 00000046 UTL default transPullupInterval 2
09/02 16:45:00.244299 00000046 UTL default mqRebalanceInterval 2
09/02 16:45:00.244314 00000046 UTL default ttlUnit 86400
09/02 16:45:00.244329 00000046 UTL default ttlPushInterval 86400
09/02 16:45:00.244352 00000046 UTL default udf 1
09/02 16:45:00.244369 00000046 UTL default configDir /etc/taos
09/02 16:45:00.244384 00000046 UTL default scriptDir /etc/taos
09/02 16:45:00.244399 00000046 UTL default logDir /var/log/taos
09/02 16:45:00.244415 00000046 UTL default minimalLogDirGB 1.00
09/02 16:45:00.244438 00000046 UTL default numOfLogLines 10000000
09/02 16:45:00.244457 00000046 UTL default asyncLog 1
09/02 16:45:00.244473 00000046 UTL default logKeepDays 0
09/02 16:45:00.244509 00000046 UTL default debugFlag 0
09/02 16:45:00.244527 00000046 UTL default simDebugFlag 143
09/02 16:45:00.244545 00000046 UTL default tmrDebugFlag 131
09/02 16:45:00.244561 00000046 UTL default uDebugFlag 131
09/02 16:45:00.244590 00000046 UTL default rpcDebugFlag 131
09/02 16:45:00.244613 00000046 UTL default jniDebugFlag 131
09/02 16:45:00.244631 00000046 UTL default qDebugFlag 131
09/02 16:45:00.244652 00000046 UTL default cDebugFlag 131
09/02 16:45:00.244669 00000046 UTL default dDebugFlag 135
09/02 16:45:00.244691 00000046 UTL default vDebugFlag 135
09/02 16:45:00.244708 00000046 UTL default mDebugFlag 135
09/02 16:45:00.244726 00000046 UTL default wDebugFlag 135
09/02 16:45:00.244744 00000046 UTL default sDebugFlag 135
09/02 16:45:00.244760 00000046 UTL default tsdbDebugFlag 131
09/02 16:45:00.244782 00000046 UTL default tqDebugFlag 135
09/02 16:45:00.244800 00000046 UTL default fsDebugFlag 135
09/02 16:45:00.244816 00000046 UTL default udfDebugFlag 135
09/02 16:45:00.244833 00000046 UTL default smaDebugFlag 135
09/02 16:45:00.244850 00000046 UTL default idxDebugFlag 135
09/02 16:45:00.244870 00000046 UTL default tdbDebugFlag 131
09/02 16:45:00.244889 00000046 UTL default metaDebugFlag 135
09/02 16:45:00.244905 00000046 UTL default timezone Etc/UTC (CST, +0800)
09/02 16:45:00.244922 00000046 UTL default locale en_US.UTF-8
09/02 16:45:00.244938 00000046 UTL default charset UTF-8
09/02 16:45:00.244961 00000046 UTL default enableCoreFile 1
09/02 16:45:00.244979 00000046 UTL default numOfCores 2.00
09/02 16:45:00.244997 00000046 UTL default openMax 1048576
09/02 16:45:00.245013 00000046 UTL default streamMax 16
09/02 16:45:00.245028 00000046 UTL default pageSizeKB 4
09/02 16:45:00.245050 00000046 UTL default totalMemoryKB 3762932
09/02 16:45:00.245074 00000046 UTL default os sysname Linux
09/02 16:45:00.245102 00000046 UTL default os nodename taos
09/02 16:45:00.245124 00000046 UTL default os release 5.10.0-14-amd64
09/02 16:45:00.245152 00000046 UTL default os version #1 SMP Debian 5.10.113-1 (2022-04-29)
09/02 16:45:00.245176 00000046 UTL default os machine x86_64
09/02 16:45:00.245199 00000046 UTL default version 3.0.0.1
09/02 16:45:00.245221 00000046 UTL default compatible_version 3.0.0.0
09/02 16:45:00.245244 00000046 UTL default gitinfo aca3d913544676c1c89c0ec732d89aed9554ea1f
09/02 16:45:00.245267 00000046 UTL default buildinfo Built at 2022-08-23 10:05
09/02 16:45:00.245289 00000046 UTL =================================================================
09/02 16:45:00.247034 00000046 DND start to init dnode env
09/02 16:45:00.247105 00000046 DND dnode will run in single-process mode
09/02 16:45:00.247749 00000046 DND dnode env is initialized
09/02 16:45:00.247843 00000046 DND start to init service
09/02 16:45:00.247913 00000046 DND node:dnode, start to open
09/02 16:45:00.247926 00000046 UTL worker:dnode-mgmt is initialized, min:1 max:1
09/02 16:45:00.248123 00000052 UDF start to init udfd
09/02 16:45:00.248772 00000052 UDF ERROR can not spawn udfd. path: taosd/udfd, error: no such file or directory
09/02 16:45:00.248958 00000046 UDF udfd is cleaned up after spawn err
09/02 16:45:00.248971 00000046 DND ERROR failed to start udfd
09/02 16:45:00.248985 00000046 DND node:dnode, has been opened
09/02 16:45:00.249006 00000046 DND node:mnode, start to open
09/02 16:45:00.249067 00000046 WAL wal module is initialized, rsetId:3
09/02 16:45:00.249334 00000046 DND mnode start to open
09/02 16:45:00.250804 00000046 MND start to open mnode sync, standby:0
09/02 16:45:00.250984 00000046 SYN vgId:1, sync node use snapshot
09/02 16:45:00.251036 00000046 UTL worker:mnode-query is initialized, min:4 max:4
09/02 16:45:00.251770 00000046 UTL worker:mnode-fetch is initialized, min:1 max:1
09/02 16:45:00.251834 00000046 UTL worker:mnode-read is initialized, min:1 max:1
09/02 16:45:00.251891 00000046 UTL worker:mnode-write is initialized, min:1 max:1
09/02 16:45:00.251959 00000046 UTL worker:mnode-sync is initialized, min:1 max:1
09/02 16:45:00.252027 00000046 UTL worker:mnode-monitor is initialized, min:1 max:1
09/02 16:45:00.252095 00000046 DND node:mnode, has been opened
09/02 16:45:00.252136 00000046 DND node:vnode, start to open
09/02 16:45:00.252298 00000046 UTL worker:vnode-query is initialized, min:4 max:4
09/02 16:45:00.252309 00000046 UTL worker:vnode-stream is initialized, min:4 max:4
09/02 16:45:00.252314 00000046 UTL worker:vnode-mgmt is initialized, min:1 max:1
09/02 16:45:00.252348 00000046 UTL worker:vnode-monitor is initialized, min:1 max:1
09/02 16:45:00.252509 00000046 DND open 2 vnodes with 1 threads
09/02 16:45:00.254967 00000071 SYN vgId:2, sync node use snapshot
09/02 16:45:00.261630 00000071 SYN vgId:3, sync node use snapshot
09/02 16:45:00.261871 00000046 DND successfully opened 2 vnodes
09/02 16:45:00.261962 00000046 UDF get dnodeId:1 from env, pipe path:/var/lib/taos/.udfd.sock.1
09/02 16:45:00.262115 00000046 UDF udfc initialized
09/02 16:45:00.262127 00000046 DND node:vnode, has been opened
09/02 16:45:00.265577 00000046 MND trans:6, execute in mnode which not leader
09/02 16:45:00.265636 00000046 MND trans:6, execute in mnode which not leader
09/02 16:45:00.265726 00000046 MND trans:7, execute in mnode which not leader
09/02 16:45:00.265777 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.265853 00000046 MND trans:7, execute in mnode which not leader
09/02 16:45:00.266001 00000046 MND trans:8, execute in mnode which not leader
09/02 16:45:00.266024 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.266119 00000046 MND trans:8, execute in mnode which not leader
09/02 16:45:00.266254 00000046 MND trans:9, execute in mnode which not leader
09/02 16:45:00.266278 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.266345 00000046 MND trans:9, execute in mnode which not leader
09/02 16:45:00.266453 00000046 MND trans:10, execute in mnode which not leader
09/02 16:45:00.266475 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.266536 00000046 MND trans:10, execute in mnode which not leader
09/02 16:45:00.266643 00000046 MND trans:11, execute in mnode which not leader
09/02 16:45:00.266665 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.266729 00000046 MND trans:11, execute in mnode which not leader
09/02 16:45:00.266847 00000046 MND trans:12, execute in mnode which not leader
09/02 16:45:00.266865 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.267138 00000046 MND mnode sync restore finished, and will handle outstanding transactions
09/02 16:45:00.267167 00000046 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014
09/02 16:45:00.271346 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:1, term:1, msg-index:1, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.271577 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:2, term:1, msg-index:2, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.271622 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:3, term:1, msg-index:3, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.271664 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:4, term:1, msg-index:4, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.271698 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:5, term:1, msg-index:5, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.271730 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:6, term:1, msg-index:6, weak:0, code:0, state:102 leader, type:drop-table
09/02 16:45:00.271764 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:7, term:1, msg-index:7, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.271797 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:8, term:1, msg-index:8, weak:0, code:0, state:102 leader, type:drop-table
09/02 16:45:00.271831 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:9, term:1, msg-index:9, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.271866 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:10, term:1, msg-index:10, weak:0, code:0, state:102 leader, type:drop-table
09/02 16:45:00.271957 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:11, term:1, msg-index:11, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.271993 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:12, term:1, msg-index:12, weak:0, code:0, state:102 leader, type:drop-table
09/02 16:45:00.272029 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:13, term:1, msg-index:13, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.272094 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:15, term:2, msg-index:15, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272158 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:17, term:3, msg-index:17, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272215 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:19, term:4, msg-index:19, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272276 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:21, term:5, msg-index:21, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272333 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:23, term:6, msg-index:23, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272388 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:25, term:7, msg-index:25, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272447 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:27, term:8, msg-index:27, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272507 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:29, term:9, msg-index:29, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272563 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:31, term:10, msg-index:31, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272617 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:33, term:11, msg-index:33, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272677 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:35, term:12, msg-index:35, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272736 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:37, term:13, msg-index:37, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272788 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:39, term:14, msg-index:39, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.272846 00000046 VND vgId:2, commit-cb is excuted, fsm:0x7f39c012d550, index:41, term:15, msg-index:41, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.276128 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:1, term:1, msg-index:1, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.276196 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:2, term:1, msg-index:2, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.276248 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:3, term:1, msg-index:3, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.276282 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:4, term:1, msg-index:4, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.276315 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:5, term:1, msg-index:5, weak:0, code:0, state:102 leader, type:vnode-create-stb
09/02 16:45:00.276369 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:6, term:1, msg-index:6, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.276407 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:7, term:1, msg-index:7, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276449 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:8, term:1, msg-index:8, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276495 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:9, term:1, msg-index:9, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276535 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:10, term:1, msg-index:10, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276574 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:11, term:1, msg-index:11, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276605 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:12, term:1, msg-index:12, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276633 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:13, term:1, msg-index:13, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276666 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:14, term:1, msg-index:14, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276703 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:15, term:1, msg-index:15, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276747 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:16, term:1, msg-index:16, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276782 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:17, term:1, msg-index:17, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276817 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:18, term:1, msg-index:18, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.276854 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:19, term:1, msg-index:19, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276889 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:20, term:1, msg-index:20, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.276921 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:21, term:1, msg-index:21, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.276955 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:22, term:1, msg-index:22, weak:0, code:0, state:102 leader, type:create-table
09/02 16:45:00.276989 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:23, term:1, msg-index:23, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277028 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:24, term:1, msg-index:24, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277062 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:25, term:1, msg-index:25, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277095 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:26, term:1, msg-index:26, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277135 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:27, term:1, msg-index:27, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277176 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:28, term:1, msg-index:28, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277210 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:29, term:1, msg-index:29, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277245 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:30, term:1, msg-index:30, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277281 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:31, term:1, msg-index:31, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277315 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:32, term:1, msg-index:32, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277360 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:33, term:1, msg-index:33, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277394 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:34, term:1, msg-index:34, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277428 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:35, term:1, msg-index:35, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277465 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:36, term:1, msg-index:36, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277504 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:37, term:1, msg-index:37, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277552 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:38, term:1, msg-index:38, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277591 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:39, term:1, msg-index:39, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277642 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:40, term:1, msg-index:40, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277682 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:41, term:1, msg-index:41, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277724 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:42, term:1, msg-index:42, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277761 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:43, term:1, msg-index:43, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277796 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:44, term:1, msg-index:44, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277832 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:45, term:1, msg-index:45, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277866 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:46, term:1, msg-index:46, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277903 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:47, term:1, msg-index:47, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277937 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:48, term:1, msg-index:48, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.277974 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:49, term:1, msg-index:49, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278016 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:50, term:1, msg-index:50, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278051 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:51, term:1, msg-index:51, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278090 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:52, term:1, msg-index:52, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278126 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:53, term:1, msg-index:53, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278168 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:54, term:1, msg-index:54, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278207 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:55, term:1, msg-index:55, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278249 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:56, term:1, msg-index:56, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278293 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:57, term:1, msg-index:57, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278328 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:58, term:1, msg-index:58, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278371 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:59, term:1, msg-index:59, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278407 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:60, term:1, msg-index:60, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278444 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:61, term:1, msg-index:61, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278478 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:62, term:1, msg-index:62, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278514 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:63, term:1, msg-index:63, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278554 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:64, term:1, msg-index:64, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278588 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:65, term:1, msg-index:65, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278634 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:66, term:1, msg-index:66, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278668 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:67, term:1, msg-index:67, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278703 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:68, term:1, msg-index:68, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278739 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:69, term:1, msg-index:69, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278775 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:70, term:1, msg-index:70, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278810 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:71, term:1, msg-index:71, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278863 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:72, term:1, msg-index:72, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278899 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:73, term:1, msg-index:73, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278933 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:74, term:1, msg-index:74, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.278969 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:75, term:1, msg-index:75, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279009 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:76, term:1, msg-index:76, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279047 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:77, term:1, msg-index:77, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279081 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:78, term:1, msg-index:78, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279118 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:79, term:1, msg-index:79, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279157 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:80, term:1, msg-index:80, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279198 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:81, term:1, msg-index:81, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279237 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:82, term:1, msg-index:82, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279271 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:83, term:1, msg-index:83, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279305 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:84, term:1, msg-index:84, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279367 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:85, term:1, msg-index:85, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279405 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:86, term:1, msg-index:86, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279443 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:87, term:1, msg-index:87, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.279577 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:88, term:1, msg-index:88, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281068 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:89, term:1, msg-index:89, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281143 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:90, term:1, msg-index:90, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281235 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:91, term:1, msg-index:91, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281378 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:92, term:1, msg-index:92, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281461 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:93, term:1, msg-index:93, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281588 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:94, term:1, msg-index:94, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281861 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:95, term:1, msg-index:95, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.281936 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:96, term:1, msg-index:96, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282004 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:97, term:1, msg-index:97, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282073 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:98, term:1, msg-index:98, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282155 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:99, term:1, msg-index:99, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282224 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:100, term:1, msg-index:100, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282293 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:101, term:1, msg-index:101, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282357 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:102, term:1, msg-index:102, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282440 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:103, term:1, msg-index:103, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282529 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:104, term:1, msg-index:104, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282598 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:105, term:1, msg-index:105, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282680 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:106, term:1, msg-index:106, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282751 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:107, term:1, msg-index:107, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282850 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:108, term:1, msg-index:108, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282922 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:109, term:1, msg-index:109, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.282998 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:110, term:1, msg-index:110, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283068 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:111, term:1, msg-index:111, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283144 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:112, term:1, msg-index:112, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283221 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:113, term:1, msg-index:113, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283303 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:114, term:1, msg-index:114, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283373 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:115, term:1, msg-index:115, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283511 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:116, term:1, msg-index:116, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283580 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:117, term:1, msg-index:117, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283667 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:118, term:1, msg-index:118, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283738 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:119, term:1, msg-index:119, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283805 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:120, term:1, msg-index:120, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283869 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:121, term:1, msg-index:121, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.283942 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:122, term:1, msg-index:122, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284010 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:123, term:1, msg-index:123, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284082 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:124, term:1, msg-index:124, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284156 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:125, term:1, msg-index:125, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284224 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:126, term:1, msg-index:126, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284288 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:127, term:1, msg-index:127, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284359 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:128, term:1, msg-index:128, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284439 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:129, term:1, msg-index:129, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284509 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:130, term:1, msg-index:130, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284576 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:131, term:1, msg-index:131, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284644 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:132, term:1, msg-index:132, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284714 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:133, term:1, msg-index:133, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284782 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:134, term:1, msg-index:134, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284848 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:135, term:1, msg-index:135, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284925 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:136, term:1, msg-index:136, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.284994 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:137, term:1, msg-index:137, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285062 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:138, term:1, msg-index:138, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285129 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:139, term:1, msg-index:139, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285207 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:140, term:1, msg-index:140, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285287 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:141, term:1, msg-index:141, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285353 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:142, term:1, msg-index:142, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285418 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:143, term:1, msg-index:143, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285489 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:144, term:1, msg-index:144, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285554 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:145, term:1, msg-index:145, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285619 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:146, term:1, msg-index:146, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285709 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:147, term:1, msg-index:147, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285789 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:148, term:1, msg-index:148, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285864 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:149, term:1, msg-index:149, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285930 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:150, term:1, msg-index:150, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.285996 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:151, term:1, msg-index:151, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286078 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:152, term:1, msg-index:152, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286144 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:153, term:1, msg-index:153, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286220 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:154, term:1, msg-index:154, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286286 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:155, term:1, msg-index:155, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286352 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:156, term:1, msg-index:156, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286424 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:157, term:1, msg-index:157, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286489 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:158, term:1, msg-index:158, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286557 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:159, term:1, msg-index:159, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286638 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:160, term:1, msg-index:160, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286732 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:161, term:1, msg-index:161, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286802 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:162, term:1, msg-index:162, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286871 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:163, term:1, msg-index:163, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.286937 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:164, term:1, msg-index:164, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287022 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:165, term:1, msg-index:165, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287092 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:166, term:1, msg-index:166, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287185 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:167, term:1, msg-index:167, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287253 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:168, term:1, msg-index:168, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287318 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:169, term:1, msg-index:169, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287383 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:170, term:1, msg-index:170, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287479 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:171, term:1, msg-index:171, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287554 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:172, term:1, msg-index:172, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287627 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:173, term:1, msg-index:173, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287705 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:174, term:1, msg-index:174, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287773 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:175, term:1, msg-index:175, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287841 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:176, term:1, msg-index:176, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287912 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:177, term:1, msg-index:177, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.287994 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:178, term:1, msg-index:178, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288063 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:179, term:1, msg-index:179, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288128 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:180, term:1, msg-index:180, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288222 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:181, term:1, msg-index:181, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288297 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:182, term:1, msg-index:182, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288364 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:183, term:1, msg-index:183, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288430 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:184, term:1, msg-index:184, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288497 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:185, term:1, msg-index:185, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288593 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:186, term:1, msg-index:186, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288664 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:187, term:1, msg-index:187, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288731 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:188, term:1, msg-index:188, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288798 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:189, term:1, msg-index:189, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288870 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:190, term:1, msg-index:190, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.288937 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:191, term:1, msg-index:191, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289002 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:192, term:1, msg-index:192, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289073 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:193, term:1, msg-index:193, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289163 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:194, term:1, msg-index:194, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289258 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:195, term:1, msg-index:195, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289328 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:196, term:1, msg-index:196, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289416 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:197, term:1, msg-index:197, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289490 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:198, term:1, msg-index:198, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289557 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:199, term:1, msg-index:199, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289623 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:200, term:1, msg-index:200, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289770 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:201, term:1, msg-index:201, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289841 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:202, term:1, msg-index:202, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289903 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:203, term:1, msg-index:203, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.289967 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:204, term:1, msg-index:204, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.290053 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:205, term:1, msg-index:205, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.290122 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:206, term:1, msg-index:206, weak:0, code:0, state:102 leader, type:submit
09/02 16:45:00.290218 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:208, term:2, msg-index:208, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290303 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:210, term:3, msg-index:210, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290376 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:212, term:4, msg-index:212, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290469 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:214, term:5, msg-index:214, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290540 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:216, term:6, msg-index:216, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290612 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:218, term:7, msg-index:218, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290695 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:220, term:8, msg-index:220, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290782 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:222, term:9, msg-index:222, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290850 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:224, term:10, msg-index:224, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.290919 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:226, term:11, msg-index:226, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.291002 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:228, term:12, msg-index:228, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.291082 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:230, term:13, msg-index:230, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.291181 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:232, term:14, msg-index:232, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.291253 00000046 VND vgId:3, commit-cb is excuted, fsm:0x7f39c0315620, index:234, term:15, msg-index:234, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb
09/02 16:45:00.291326 00000046 DND TDengine initialized successfully
09/02 16:45:01.262958 00000063 MND dnode:1, mnode syncstate from error to leader
09/02 16:45:01.262986 00000063 MND dnode:1, from offline to online, memory avail:3451140916 total:3853242368 cores:2.00
09/02 16:45:01.272171 00000069 VND vgId:2, start to commit, commit ID:1 version:43
09/02 16:45:01.275541 00000069 VND vgId:2, vnode info is saved, fname:/var/lib/taos/vnode/vnode2/vnode_tmp.json
taosd: /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c:1914: tsdbCmprData: Assertion `szIn > 0 && ppOut' failed.
Aborted (core dumped)
root@VM-16-2-debian:~#
@yu285 may be the emqx is writing data always? so it occured if i drop database at the same time?
it is good that we reproduced it , but we need to get the core file dumped by Linux kernel .
so could you configure as below :
1、ulimit -c unlimited
2、sudo sysctl -w kernel.core_pattern='/corefile/core-%e-%p'
echo '/corefile/core_%e-%p' | sudo tee /proc/sys/kernel/core_pattern
3、"sysctl kernel.core_pattern" to check if configuration works.
4.sudo mkdir /corefile && chmod 777 /corefile
before above operation all done . you should stop the emqx writing program .
then rerun emqx , you could see a core file under /corefile .
it is good that we reproduced it , but we need to get the core file dumped by Linux kernel .
so could you configure as below :
1、ulimit -c unlimited 2、sudo sysctl -w kernel.core_pattern='/corefile/core-%e-%p' echo '/corefile/core_%e-%p' | sudo tee /proc/sys/kernel/core_pattern 3、"sysctl kernel.core_pattern" to check if configuration works. 4.sudo mkdir /corefile && chmod 777 /corefilebefore above operation all done . you should stop the emqx writing program .
then rerun emqx , you could see a core file under /corefile .
any other opertaions for docker ?
it is good that we reproduced it , but we need to get the core file dumped by Linux kernel . so could you configure as below :
1、ulimit -c unlimited 2、sudo sysctl -w kernel.core_pattern='/corefile/core-%e-%p' echo '/corefile/core_%e-%p' | sudo tee /proc/sys/kernel/core_pattern 3、"sysctl kernel.core_pattern" to check if configuration works. 4.sudo mkdir /corefile && chmod 777 /corefilebefore above operation all done . you should stop the emqx writing program . then rerun emqx , you could see a core file under /corefile .
any other opertaions for docker ?
that is why I suggest you use physical machine ,because the core file in docker is not readable even though you mount local filesystem there .
if no physical machine environment
1.I think you can do above operations inside docker. 2.then reproduced problem. 3.backup the datafile then remove them 4. restart docker again 5.cd to /corefile to check if corefile dumped .
it is good that we reproduced it , but we need to get the core file dumped by Linux kernel . so could you configure as below :
1、ulimit -c unlimited 2、sudo sysctl -w kernel.core_pattern='/corefile/core-%e-%p' echo '/corefile/core_%e-%p' | sudo tee /proc/sys/kernel/core_pattern 3、"sysctl kernel.core_pattern" to check if configuration works. 4.sudo mkdir /corefile && chmod 777 /corefilebefore above operation all done . you should stop the emqx writing program . then rerun emqx , you could see a core file under /corefile .
any other opertaions for docker ?
that is why I suggest you use physical machine ,because the core file in docker is not readable even though you mount local filesystem there .
if no physical machine environment
1.I think you can do above operations inside docker. 2.then reproduced problem. 3.backup the datafile then remove them 4. restart docker again 5.cd to /corefile to check if corefile dumped .
WTF?
https://docs.taosdata.com/get-started/package/
wget https://www.taosdata.com/assets-download/3.0/TDengine-server-3.0.0.1-Linux-x64.deb
dpkg -i TDengine-server-3.0.0.1-Linux-x64.deb
systemctl start taosd
systemctl status taosd
● taosd.service - TDengine server service
Loaded: loaded (/etc/systemd/system/taosd.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-09-02 17:15:53 CST; 19min ago
Process: 515785 ExecStartPre=/usr/local/taos/bin/startPre.sh (code=exited, status=0/SUCCESS)
Main PID: 515790 (taosd)
Tasks: 52 (limit: 4383)
Memory: 26.2M
CPU: 6.432s
CGroup: /system.slice/taosd.service
├─515790 /usr/bin/taosd
└─515797 /usr/bin/udfd -c /etc/taos
Sep 02 17:15:53 VM-16-2-debian systemd[1]: Starting TDengine server service...
Sep 02 17:15:53 VM-16-2-debian systemd[1]: Started TDengine server service.
root@VM-16-2-debian:~# taos -k
2: service ok
root@VM-16-2-debian:~#
root@VM-16-2-debian:~# taos
Welcome to the TDengine Command Line Interface, Client Version:3.0.0.1
Copyright (c) 2022 by TDengine, all rights reserved.
Server is Community Edition.
taos> exit;
root@VM-16-2-debian:~# curl -u root:taosdata -d 'show databases' http://localhost:6041/rest/sql
curl: (7) Failed to connect to localhost port 6041: Connection refused
???
you
it is good that we reproduced it , but we need to get the core file dumped by Linux kernel . so could you configure as below :
1、ulimit -c unlimited 2、sudo sysctl -w kernel.core_pattern='/corefile/core-%e-%p' echo '/corefile/core_%e-%p' | sudo tee /proc/sys/kernel/core_pattern 3、"sysctl kernel.core_pattern" to check if configuration works. 4.sudo mkdir /corefile && chmod 777 /corefilebefore above operation all done . you should stop the emqx writing program . then rerun emqx , you could see a core file under /corefile .
any other opertaions for docker ?
that is why I suggest you use physical machine ,because the core file in docker is not readable even though you mount local filesystem there . if no physical machine environment 1.I think you can do above operations inside docker. 2.then reproduced problem. 3.backup the datafile then remove them 4. restart docker again 5.cd to /corefile to check if corefile dumped .
WTF?
https://docs.taosdata.com/get-started/package/
wget https://www.taosdata.com/assets-download/3.0/TDengine-server-3.0.0.1-Linux-x64.deb dpkg -i TDengine-server-3.0.0.1-Linux-x64.deb systemctl start taosd systemctl status taosd ● taosd.service - TDengine server service Loaded: loaded (/etc/systemd/system/taosd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-09-02 17:15:53 CST; 19min ago Process: 515785 ExecStartPre=/usr/local/taos/bin/startPre.sh (code=exited, status=0/SUCCESS) Main PID: 515790 (taosd) Tasks: 52 (limit: 4383) Memory: 26.2M CPU: 6.432s CGroup: /system.slice/taosd.service ├─515790 /usr/bin/taosd └─515797 /usr/bin/udfd -c /etc/taos Sep 02 17:15:53 VM-16-2-debian systemd[1]: Starting TDengine server service... Sep 02 17:15:53 VM-16-2-debian systemd[1]: Started TDengine server service. root@VM-16-2-debian:~# taos -k 2: service ok root@VM-16-2-debian:~# root@VM-16-2-debian:~# taos Welcome to the TDengine Command Line Interface, Client Version:3.0.0.1 Copyright (c) 2022 by TDengine, all rights reserved. Server is Community Edition. taos> exit; root@VM-16-2-debian:~# curl -u root:taosdata -d 'show databases' http://localhost:6041/rest/sql curl: (7) Failed to connect to localhost port 6041: Connection refused ???
you didn't run "systemctl start taosadapter" to enable REST service
@yu285
taos> drop database synron_cloud_store;
DB error: Unable to establish connection (0.169818s)
taos>
taos> select * from client_plan_status;
DB error: Fail to get table info, error: Unable to establish connection (0.076823s)
taos> exit
root@VM-16-2-debian:~# taos -k
0: unavailableroot@VM-16-2-debian:~# ls -al /corefile/
total 47976
drwxrwxrwx 2 root root 4096 Sep 2 17:43 .
drwxr-xr-x 21 root root 4096 Sep 2 17:44 ..
-rw------- 1 root root 553877504 Sep 2 17:43 core_vnode-mgmt-521336
-rw------- 1 root root 516825088 Sep 2 17:43 core_vnode-mgmt-522190
-rw------- 1 root root 516825088 Sep 2 17:43 core_vnode-mgmt-522267
-rw------- 1 root root 516825088 Sep 2 17:43 core_vnode-mgmt-522333
root@VM-16-2-debian:~#
it's very large? how to send it to you
please install a gdb with "sudo apt-get install gdb"
then "sudo gdb taosd core_vnode-mgmt-521336"
then type "bt" ,show us capture
And please also describe what operations you've done to reproduce it step by step
please install a gdb with "sudo apt-get install gdb"
then "sudo gdb taosd core_vnode-mgmt-521336"
then type "bt" ,show us capture
Core was generated by `/usr/bin/taosd'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f5726ffd700 (LWP 521363))]
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f573bd0d5ef in __GI_abort () at abort.c:100
#2 0x00007f573bd0d40f in __assert_fail_base (fmt=0x7f573be76128 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x55a1f2cecf00 "szIn > 0 && ppOut",
file=0x55a1f2ceca90 "/home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c", line=1914, function=<optimized out>) at assert.c:92
#3 0x00007f573bd1c662 in __GI___assert_fail (assertion=0x55a1f2cecf00 "szIn > 0 && ppOut",
file=0x55a1f2ceca90 "/home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c", line=1914,
function=0x55a1f2ced218 <__PRETTY_FUNCTION__.21987> "tsdbCmprData") at assert.c:101
#4 0x000055a1f2961edf in tsdbCmprData (pIn=0x0, szIn=0, type=10 '\n', cmprAlg=2 '\002', ppOut=0x7f56f8086088, nOut=190, szOut=0x7f5726ffc294, ppBuf=0x7f56f8086098)
at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c:1914
#5 0x000055a1f2962a04 in tsdbCmprColData (pColData=0x7f56f8085f50, cmprAlg=2 '\002', pBlockCol=0x7f5726ffc280, ppOut=0x7f56f8086088, nOut=180, ppBuf=0x7f56f8086098)
at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c:2043
#6 0x000055a1f295efa1 in tCmprBlockData (pBlockData=0x7f5726ffc6f0, cmprAlg=2 '\002', ppOut=0x0, szOut=0x0, aBuf=0x7f56f8086088, aBufN=0x7f5726ffc350)
at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbUtil.c:1531
#7 0x000055a1f295263b in tsdbWriteBlockData (pWriter=0x7f56f8085fc0, pBlockData=0x7f5726ffc6f0, pBlkInfo=0x7f5726ffc430, pSmaInfo=0x0, cmprAlg=2 '\002', toLast=1 '\001')
at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c:1438
#8 0x000055a1f293e904 in tsdbCommitLastBlock (pCommitter=0x7f5726ffc520) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbCommit.c:580
#9 0x000055a1f2941085 in tsdbCommitFileData (pCommitter=0x7f5726ffc520) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbCommit.c:1238
#10 0x000055a1f29415e5 in tsdbCommitData (pCommitter=0x7f5726ffc520) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbCommit.c:1368
#11 0x000055a1f293cc06 in tsdbCommit (pTsdb=0x7f57203086b0) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/tsdb/tsdbCommit.c:140
#12 0x000055a1f290cf23 in vnodeCommit (pVnode=0x7f57201f32e0) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/vnd/vnodeCommit.c:272
#13 0x000055a1f29096cc in vnodeClose (pVnode=0x7f57201f32e0) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/vnode/src/vnd/vnodeOpen.c:192
#14 0x000055a1f287f3f5 in vmCloseVnode (pMgmt=0x55a1f5b8a4c0, pVnode=0x7f57203120a0) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/mgmt/mgmt_vnode/src/vmInt.c:99
#15 0x000055a1f287dee3 in vmProcessDropVnodeReq (pMgmt=0x55a1f5b8a4c0, pMsg=0x7f56f8010b10)
at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c:322
#16 0x000055a1f2880eaa in vmProcessMgmtQueue (pInfo=0x7f5726ffcec0, pMsg=0x7f56f8010b10)
at /home/ubuntu/workroom/jenkins/TDinternal/community/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c:47
#17 0x000055a1f2c7a512 in tQWorkerThreadFp (worker=0x55a1f5b8c060) at /home/ubuntu/workroom/jenkins/TDinternal/community/source/util/src/tworker.c:90
#18 0x00007f573bec6ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#19 0x00007f573bde5def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
And please also describe what operations you've done to reproduce it step by step
- start taosdb
- create database XX
- create tables
- emqx writing data per 30s
- select * from tableX1
- select * from tableX2
- drop databse XX
- then coredump
Thank you for your cooperation.
if anything else needed we will contact you here.
if bug is fixed ,we will inform you here either .
This issue has been fixed , u can use develop branch to do a test . or wait us to release the new version.