platform icon indicating copy to clipboard operation
platform copied to clipboard

dashmate autostarts and fails on system restart

Open dashameter opened this issue 2 years ago • 6 comments

restarting the system without first manually stopping dashmate results in the dashd containers falling over and tenderdash melting the cpu:

CONTAINER ID   IMAGE                           COMMAND                  CREATED       STATUS                          PORTS                                                                                                   NAMES
cf0f942d2a8a   envoyproxy/envoy:v1.16-latest   "/docker-entrypoint.…"   3 hours ago   Up 25 minutes                   0.0.0.0:3000->10000/tcp, :::3000->10000/tcp, 0.0.0.0:3010->50051/tcp, :::3010->50051/tcp                dash_masternode_local_1_dapi_envoy_1
2c4bce954167   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_1_dapi_api_1
2dea21632e0b   envoyproxy/envoy:v1.16-latest   "/docker-entrypoint.…"   3 hours ago   Up 25 minutes                   0.0.0.0:3200->10000/tcp, :::3200->10000/tcp, 0.0.0.0:3210->50051/tcp, :::3210->50051/tcp                dash_masternode_local_3_dapi_envoy_1
52a58c2f3a6b   envoyproxy/envoy:v1.16-latest   "/docker-entrypoint.…"   3 hours ago   Up 25 minutes                   0.0.0.0:3100->10000/tcp, :::3100->10000/tcp, 0.0.0.0:3110->50051/tcp, :::3110->50051/tcp                dash_masternode_local_2_dapi_envoy_1
9562bf05a20c   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_3_dapi_api_1
75ce1c9afc4b   dashpay/tenderdash:0.6.0        "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   0.0.0.0:26656->26656/tcp, :::26656->26656/tcp, 127.0.0.1:26657->26657/tcp, 26660/tcp                    dash_masternode_local_1_drive_tenderdash_1
02d90b813b18   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_2_dapi_api_1
44a255a284f9   dashpay/tenderdash:0.6.0        "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   26656-26657/tcp, 26660/tcp, 0.0.0.0:26856->26856/tcp, :::26856->26856/tcp, 127.0.0.1:26857->26857/tcp   dash_masternode_local_3_drive_tenderdash_1
7fd001ac1646   dashpay/tenderdash:0.6.0        "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   26656-26657/tcp, 26660/tcp, 0.0.0.0:26756->26756/tcp, :::26756->26756/tcp, 127.0.0.1:26757->26757/tcp   dash_masternode_local_2_drive_tenderdash_1
e392afd8fb7c   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_3_dapi_tx_filter_stream_1
b4f70b7a5862   dashpay/sentinel:1.6.0          "/bin/sh -c /run.sh"     3 hours ago   Up 25 minutes                                                                                                                           dash_masternode_local_3_sentinel_1
7463c27b39e3   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_2_dapi_tx_filter_stream_1
8c359ce7cd8b   dashpay/sentinel:1.6.0          "/bin/sh -c /run.sh"     3 hours ago   Up 25 minutes                                                                                                                           dash_masternode_local_2_sentinel_1
a7dd03c63fa7   dashpay/dapi:0.21               "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   2500-2501/tcp, 2510/tcp                                                                                 dash_masternode_local_1_dapi_tx_filter_stream_1
fb32f243fee5   dashpay/sentinel:1.6.0          "/bin/sh -c /run.sh"     3 hours ago   Up 25 minutes                                                                                                                           dash_masternode_local_1_sentinel_1
6ac2e68a3652   dashpay/dashd:0.18.0.0-beta1    "/docker-entrypoint.…"   3 hours ago   Restarting (1) 42 seconds ago                                                                                                           dash_masternode_local_2_core_1
47f396a261ee   mongo:4.2                       "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   27017/tcp                                                                                               dash_masternode_local_2_drive_mongodb_1
43bd24eedc16   mongo:4.2                       "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   27017/tcp                                                                                               dash_masternode_local_3_drive_mongodb_1
36175b229d2a   dashpay/dashd:0.18.0.0-beta1    "/docker-entrypoint.…"   3 hours ago   Restarting (1) 43 seconds ago                                                                                                           dash_masternode_local_3_core_1
9877df2e9929   mongo:4.2                       "docker-entrypoint.s…"   3 hours ago   Up 25 minutes                   27017/tcp                                                                                               dash_masternode_local_1_drive_mongodb_1
d881dfd6fe95   dashpay/dashd:0.18.0.0-beta1    "/docker-entrypoint.…"   3 hours ago   Restarting (1) 46 seconds ago   

The dashd container log is:

2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) init message: Verifying blocks...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) : The block database contains a block which appears to be from the future. This may be due to your computer's date and time being set incorrectly. Only rebuild the block database if you are sure that your computer's date and time are correct.
Please restart with -reindex or -reindex-chainstate to recover.
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Aborted block database rebuild. Exiting.
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Interrupting HTTP server                                                                      
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Interrupting HTTP RPC server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Interrupting RPC
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) PrepareShutdown: In progress...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopping HTTP RPC server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Unregistering HTTP handler for / (exactmatch 1)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Unregistering HTTP handler for /wallet/ (exactmatch 0)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopping RPC
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) RPC stopped.
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopping HTTP server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Waiting for HTTP worker threads to exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Exited http event loop
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Waiting for HTTP event thread to exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Stopped HTTP server
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) cl-schdlr thread exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) scheduler thread exit
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Generated table #127@0: 1708 keys, 110723 bytes
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Compacted 4@0 + 1@1 files => 110723 bytes
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: compacted to: files[ 0 1 0 0 0 0 0 ]                   
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #114
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #122
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #119
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #125
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) leveldb: Delete type=2 #116
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Writing final batch of 0.00 MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Committed 0 changed transaction outputs (out of 0) to coin database...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=evodb, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block and undo data to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write block index to disk completed (0.00ms)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) started
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Writing final batch of 0.00 MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=chainstate, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Committed 0 changed transaction outputs (out of 0) to coin database...
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) WriteBatch memory usage: db=evodb, before=0.0MiB, after=0.0MiB
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) FlushStateToDisk: write coins cache to disk (0 coins, 0kB) completed (0.00s)
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notification interface
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubhashblock at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawchainlocksig at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawtx at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawtxlock at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Shutdown notifier pubrawtxlocksig at tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) zmq: Close socket at address tcp://0.0.0.0:29998
2022-02-25T16:17:42Z (mocktime: 2022-02-25T13:06:27Z) Shutdown: done

dashameter avatar Feb 26 '22 11:02 dashameter

please recheck this with v22

dash-maverick avatar Mar 25 '22 08:03 dash-maverick

retest failed, dashd is still falling over after a system reboot

dashameter avatar Mar 27 '22 10:03 dashameter

@dashameter as a workaround for now - after the restart, kill all dashmate related docker containers. You can just go with docker kill $(docker ps -q), but have in mind that it will kill ALl containers.

After that try with dashmate start.

markin-io avatar Apr 08 '22 08:04 markin-io

Workaround blocked due to new Dashmate issue [to be raised]

SamKirby22 avatar May 06 '22 07:05 SamKirby22

image

retest failed, tenderdash sits at 125% CPU after system reboot

dashameter avatar Jun 17 '22 07:06 dashameter

System restart problem still actual for local network due to unreliable mock time logic

shumkov avatar Sep 26 '24 18:09 shumkov