celestia-node
celestia-node copied to clipboard
celestiaorg
bug: BN could take longer than 1m30s to shutdown via SIGTERM
When there is a link split between "celestia-appd" node and "celestia bridge node" (reverse tunnel connection):
Jul 15 06:55:51 celestia[1590304]: 2024-07-15T06:55:51.996Z INFO header/store store/store.go:367 new head {"height": 2267583, "hash": "434670B64F7AF16A18B858B852BE14163163FB09F151BC4CD7D4D7BA848AD102"}
Jul 15 06:56:41 celestia[1590304]: 2024-07-15T06:56:41.997Z INFO core core/listener.go:204 listener: listening stopped
Jul 15 06:56:42 celestia[1590304]: 2024-07-15T06:56:41.999Z WARN core core/listener.go:137 listener: subscriber error, resubscribing... {"err": "underlying subscription is stuck"}
Jul 15 06:56:46 celestia[1590304]: 2024-07-15T06:56:46.644Z WARN badger4 [email protected]/db.go:460 Block cache might be too small. Metrics: hit: 136710752 miss: 5972539492 keys-added: 132017714 keys-updated: 702935 keys-evicted: 132017678 cost-added: 817470580729946 cost-evicted: 817470315388995 sets-dropped: 344163 sets-rejected: 1674277784 gets-dropped: 3229628800 gets-kept: 2879375680 gets-total: 6109250244 hit-ratio: 0.02
it is impossible to restart the service via SIGTERM (Jul 15 07:01:54):
Jul 15 07:01:46 celestia[1590304]: 2024-07-15T07:01:46.643Z WARN badger4 [email protected]/db.go:461 Cache life expectancy (in seconds):
Jul 15 07:01:46 celestia[1590304]: -- Histogram:
Jul 15 07:01:46 celestia[1590304]: Min value: 0
Jul 15 07:01:46 celestia[1590304]: Max value: 8731
Jul 15 07:01:46 celestia[1590304]: Count: 131936161
Jul 15 07:01:46 celestia[1590304]: 50p: 2.00
Jul 15 07:01:46 celestia[1590304]: 75p: 2.00
Jul 15 07:01:46 celestia[1590304]: 90p: 2.00
Jul 15 07:01:46 celestia[1590304]: [0, 2) 126762286 96.08% 96.08%
Jul 15 07:01:46 celestia[1590304]: [2, 4) 2737868 2.08% 98.15%
Jul 15 07:01:46 celestia[1590304]: [4, 8) 1389161 1.05% 99.21%
Jul 15 07:01:46 celestia[1590304]: [8, 16) 623392 0.47% 99.68%
Jul 15 07:01:46 celestia[1590304]: [16, 32) 257515 0.20% 99.87%
Jul 15 07:01:46 celestia[1590304]: [32, 64) 99470 0.08% 99.95%
Jul 15 07:01:46 celestia[1590304]: [64, 128) 39114 0.03% 99.98%
Jul 15 07:01:46 celestia[1590304]: [128, 256) 17191 0.01% 99.99%
Jul 15 07:01:46 celestia[1590304]: [256, 512) 7248 0.01% 100.00%
Jul 15 07:01:46 celestia[1590304]: [512, 1024) 1884 0.00% 100.00%
Jul 15 07:01:46 celestia[1590304]: [1024, 2048) 742 0.00% 100.00%
Jul 15 07:01:46 celestia[1590304]: [2048, 4096) 233 0.00% 100.00%
Jul 15 07:01:46 celestia[1590304]: [4096, 8192) 56 0.00% 100.00%
Jul 15 07:01:46 celestia[1590304]: [8192, 16384) 1 0.00% 100.00%
Jul 15 07:01:46 celestia[1590304]: --
Jul 15 07:01:54 systemd[1]: Stopping celestia-node Bridge daemon...
Jul 15 07:01:54 celestia[1590304]: 2024-07-15T07:01:54.741Z ERROR module/p2p p2p/metrics.go:47 Error starting Prometheus metrics exporter http server: http: Server closed
Jul 15 07:01:54 celestia[1590304]: 2024-07-15T07:01:54.962Z INFO rpc rpc/server.go:108 server stopped
Jul 15 07:02:46 celestia[1590304]: 2024-07-15T07:02:46.644Z WARN badger4 [email protected]/db.go:460 Block cache might be too small. Metrics: hit: 136714326 miss: 5972582101 keys-added: 132020358 keys-updated: 702945 keys-evicted: 132020296 cost-added: 817478601041102 cost-evicted: 817478334073653 sets-dropped: 344163 sets-rejected: 1674317739 gets-dropped: 3229628800 gets-kept: 2879421952 gets-total: 6109296427 hit-ratio: 0.02
Jul 15 07:02:46 celestia[1590304]: 2024-07-15T07:02:46.644Z WARN badger4 [email protected]/db.go:461 Cache life expectancy (in seconds):
Jul 15 07:02:46 celestia[1590304]: -- Histogram:
Jul 15 07:02:46 celestia[1590304]: Min value: 0
Jul 15 07:02:46 celestia[1590304]: Max value: 8731
Jul 15 07:02:46 celestia[1590304]: Count: 131936277
Jul 15 07:02:46 celestia[1590304]: 50p: 2.00
Jul 15 07:02:46 celestia[1590304]: 75p: 2.00
Jul 15 07:02:46 celestia[1590304]: 90p: 2.00
Jul 15 07:02:46 celestia[1590304]: [0, 2) 126762380 96.08% 96.08%
Jul 15 07:02:46 celestia[1590304]: [2, 4) 2737875 2.08% 98.15%
Jul 15 07:02:46 celestia[1590304]: [4, 8) 1389166 1.05% 99.21%
Jul 15 07:02:46 celestia[1590304]: [8, 16) 623398 0.47% 99.68%
Jul 15 07:02:46 celestia[1590304]: [16, 32) 257519 0.20% 99.87%
Jul 15 07:02:46 celestia[1590304]: [32, 64) 99470 0.08% 99.95%
Jul 15 07:02:46 celestia[1590304]: [64, 128) 39114 0.03% 99.98%
Jul 15 07:02:46 celestia[1590304]: [128, 256) 17191 0.01% 99.99%
Jul 15 07:02:46 celestia[1590304]: [256, 512) 7248 0.01% 100.00%
Jul 15 07:02:46 celestia[1590304]: [512, 1024) 1884 0.00% 100.00%
Jul 15 07:02:46 celestia[1590304]: [1024, 2048) 742 0.00% 100.00%
Jul 15 07:02:46 celestia[1590304]: [2048, 4096) 233 0.00% 100.00%
Jul 15 07:02:46 celestia[1590304]: [4096, 8192) 56 0.00% 100.00%
Jul 15 07:02:46 celestia[1590304]: [8192, 16384) 1 0.00% 100.00%
Jul 15 07:02:46 celestia[1590304]: --
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: State 'stop-sigterm' timed out. Killing.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: State 'stop-sigterm' timed out. Killing.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590304 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590305 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590307 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590308 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590311 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590312 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590314 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590317 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590319 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590342 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590343 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590345 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590443 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590444 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590445 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590446 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590447 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590673 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590674 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590675 (celestia) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1590676 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1600134 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1600440 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1600441 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1600442 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1600443 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1600444 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1621716 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1623013 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1673640 (n/a) with signal SIGKILL.
Jul 15 07:03:24 systemd[1]: celestia-bridge.service: Killing process 1838697 (n/a) with signal SIGKILL.
Jul 15 07:03:29 systemd[1]: celestia-bridge.service: Main process exited, code=killed, status=9/KILL
Jul 15 07:03:29 systemd[1]: celestia-bridge.service: Failed with result 'timeout'.
Jul 15 07:03:29 systemd[1]: Stopped celestia-node Bridge daemon.
Jul 15 07:03:29 systemd[1]: celestia-bridge.service: Consumed 1w 2d 13h 46min 36.493s CPU time.
Jul 15 07:03:29 systemd[1]: Started celestia-node Bridge daemon.
Jul 15 07:03:29 celestia[2012306]: 2024-07-15T07:03:29.931Z INFO node nodebuilder/module.go:26 Accessing keyring...
Furthermore, after the restart (via systemd with SIGKILL) these errors appeared:
Jul 15 07:06:05 celestia[2012367]: 2024-07-15T07:06:05.899Z INFO badger4 [email protected]/levels.go:118 21941 tables out of 22939 opened in 48.005s
Jul 15 07:06:08 celestia[2012367]: 2024-07-15T07:06:08.895Z INFO badger4 [email protected]/levels.go:118 22836 tables out of 22939 opened in 51.002s
Jul 15 07:06:09 celestia[2012367]: 2024-07-15T07:06:09.289Z INFO badger4 [email protected]/levels.go:171 All 22939 tables opened in 51.396s
Jul 15 07:06:09 celestia[2012367]: 2024-07-15T07:06:09.472Z INFO badger4 [email protected]/discard.go:66 Discard stats nextEmptySlot: 0
Jul 15 07:06:09 celestia[2012367]: 2024-07-15T07:06:09.488Z INFO badger4 [email protected]/db.go:368 Set nextTxnTs to 1639638
Jul 15 07:06:09 celestia[2012367]: 2024-07-15T07:06:09.504Z INFO badger4 [email protected]/value.go:592 Deleting empty file: /home/celestia/.celestia-bridge-mocha-4/inverted_index//000013.vlog
Jul 15 07:06:19 celestia[2012367]: 2024-07-15T07:06:19.551Z ERROR badger4 [email protected]/iterator.go:171 Unable to read: Key: [47 104 101 97 100 101 114 115 47 48 48 48 50 66 66 55 67 53 66 57 50 49 69 50 68 69 52 54 50 52 50 70 68 52 70 53 57 69 55 69 67 69 50 55 56 67 65 55 67 55 56 70 50 53 57 53 67 70 69 70 52 65 69 53 69 48 56 70 50 66 66 69 57], Version : 192702508, meta: 66, userMeta: 0 Error: file with ID: 253 not found
Jul 15 07:06:19 celestia[2012367]: 2024-07-15T07:06:19.552Z ERROR badger4 [email protected]/iterator.go:194 Key: [47 104 101 97 100 101 114 115 47 48 48 48 50 66 66 55 67 53 66 57 50 49 69 50 68 69 52 54 50 52 50 70 68 52 70 53 57 69 55 69 67 69 50 55 56 67 65 55 67 55 56 70 50 53 57 53 67 70 69 70 52 65 69 53 69 48 56 70 50 66 66 69 57], Version : 192702508, meta: 2, userMeta: 0 valuePointer: {Fid:254 Len:26470 Offset:211221245}
Jul 15 07:06:19 celestia[2012367]: 2024-07-15T07:06:19.557Z ERROR badger4 [email protected]/iterator.go:171 Unable to read: Key: [47 104 101 97 100 101 114 115 47 48 48 48 51 51 57 66 53 56 50 48 57 65 56 54 69 49 51 57 51 53 57 55 55 55 65 52 50 53 69 57 53 67 56 49 57 52 54 56 67 50 55 70 56 56 48 51 51 57 53 54 70 57 52 48 53 49 56 57 50 66 70 51 53], Version : 193509558, meta: 66, userMeta: 0 Error: file with ID: 253 not found
Jul 15 07:06:19 celestia[2012367]: 2024-07-15T07:06:19.557Z ERROR badger4 [email protected]/iterator.go:194 Key: [47 104 101 97 100 101 114 115 47 48 48 48 51 51 57 66 53 56 50 48 57 65 56 54 69 49 51 57 51 53 57 55 55 55 65 52 50 53 69 57 53 67 56 49 57 52 54 56 67 50 55 70 56 56 48 51 51 57 53 54 70 57 52 48 53 49 56 57 50 66 70 51 53], Version : 193509558, meta: 2, userMeta: 0 valuePointer: {Fid:254 Len:26293 Offset:363430027}
Jul 15 07:06:23 celestia[2012367]: 2024-07-15T07:06:23.741Z ERROR badger4 [email protected]/iterator.go:171 Unable to read: Key: [47 104 101 97 100 101 114 115 47 48 48 48 50 66 66 55 67 53 66 57 50 49 69 50 68 69 52 54 50 52 50 70 68 52 70 53 57 69 55 69 67 69 50 55 56 67 65 55 67 55 56 70 50 53 57 53 67 70 69 70 52 65 69 53 69 48 56 70 50 66 66 69 57], Version : 192702508, meta: 66, userMeta: 0 Error: file with ID: 253 not found
Jul 15 07:06:23 celestia[2012367]: 2024-07-15T07:06:23.741Z ERROR badger4 [email protected]/iterator.go:194 Key: [47 104 101 97 100 101 114 115 47 48 48 48 50 66 66 55 67 53 66 57 50 49 69 50 68 69 52 54 50 52 50 70 68 52 70 53 57 69 55 69 67 69 50 55 56 67 65 55 67 55 56 70 50 53 57 53 67 70 69 70 52 65 69 53 69 48 56 70 50 66 66 69 57], Version : 192702508, meta: 2, userMeta: 0 valuePointer: {Fid:254 Len:26470 Offset:211221245}
Jul 15 07:06:23 celestia[2012367]: 2024-07-15T07:06:23.741Z ERROR badger4 [email protected]/iterator.go:171 Unable to read: Key: [47 104 101 97 100 101 114 115 47 48 48 48 51 51 57 66 53 56 50 48 57 65 56 54 69 49 51 57 51 53 57 55 55 55 65 52 50 53 69 57 53 67 56 49 57 52 54 56 67 50 55 70 56 56 48 51 51 57 53 54 70 57 52 48 53 49 56 57 50 66 70 51 53], Version : 193509558, meta: 66, userMeta: 0 Error: file with ID: 253 not found
Jul 15 07:06:23 celestia[2012367]: 2024-07-15T07:06:23.741Z ERROR badger4 [email protected]/iterator.go:194 Key: [47 104 101 97 100 101 114 115 47 48 48 48 51 51 57 66 53 56 50 48 57 65 56 54 69 49 51 57 51 53 57 55 55 55 65 52 50 53 69 57 53 67 56 49 57 52 54 56 67 50 55 70 56 56 48 51 51 57 53 54 70 57 52 48 53 49 56 57 50 66 70 51 53], Version : 193509558, meta: 2, userMeta: 0 valuePointer: {Fid:254 Len:26293 Offset:363430027}
Jul 15 07:06:24 celestia[2012367]: 2024-07-15T07:06:24.141Z WARN shrex-sub shrexsub/pubsub.go:83 unregistering topic {"err": "no validator for topic mocha-4/eds-sub/v0.1.0"}
Jul 15 07:06:24 celestia[2012367]: 2024-07-15T07:06:24.302Z INFO badger4 [email protected]/db.go:546 Lifetime L0 stalled for: 0s
Jul 15 07:06:24 celestia[2012367]: 2024-07-15T07:06:24.303Z INFO badger4 [email protected]/db.go:625
and they disappeared only after 4h and 30m
Jul 15 11:33:05 celestia[2012453]: 2024-07-15T11:33:05.062Z INFO header/store store/store.go:367 new head {"height": 2268971, "hash": "785D8D911B6852A81EE772D406B76CE52FECF85EE19D2F271429206A1EE0CD57"}
Jul 15 11:33:11 celestia[2012453]: 2024-07-15T11:33:11.950Z ERROR badger4 [email protected]/iterator.go:171 Unable to read: Key: [47 104 101 97 100 101 114 115 47 48 48 48 50 66 66 55 67 53 66 57 50 49 69 50 68 69 52 54 50 52 50 70 68 52 70 53 57 69 55 69 67 69 50 55 56 67 65 55 67 55 56 70 50 53 57 53 67 70 69 70 52 65 69 53 69 48 56 70 50 66 66 69 57], Version : 192702508, meta: 66, userMeta: 0 Error: file with ID: 253 not found
Jul 15 11:33:11 celestia[2012453]: 2024-07-15T11:33:11.950Z ERROR badger4 [email protected]/iterator.go:194 Key: [47 104 101 97 100 101 114 115 47 48 48 48 50 66 66 55 67 53 66 57 50 49 69 50 68 69 52 54 50 52 50 70 68 52 70 53 57 69 55 69 67 69 50 55 56 67 65 55 67 55 56 70 50 53 57 53 67 70 69 70 52 65 69 53 69 48 56 70 50 66 66 69 57], Version : 192702508, meta: 2, userMeta: 0 valuePointer: {Fid:254 Len:26470 Offset:211221245}
Jul 15 11:33:11 celestia[2012453]: 2024-07-15T11:33:11.950Z ERROR badger4 [email protected]/iterator.go:171 Unable to read: Key: [47 104 101 97 100 101 114 115 47 48 48 48 51 51 57 66 53 56 50 48 57 65 56 54 69 49 51 57 51 53 57 55 55 55 65 52 50 53 69 57 53 67 56 49 57 52 54 56 67 50 55 70 56 56 48 51 51 57 53 54 70 57 52 48 53 49 56 57 50 66 70 51 53], Version : 193509558, meta: 66, userMeta: 0 Error: file with ID: 253 not found
Jul 15 11:33:11 celestia[2012453]: 2024-07-15T11:33:11.950Z ERROR badger4 [email protected]/iterator.go:194 Key: [47 104 101 97 100 101 114 115 47 48 48 48 51 51 57 66 53 56 50 48 57 65 56 54 69 49 51 57 51 53 57 55 55 55 65 52 50 53 69 57 53 67 56 49 57 52 54 56 67 50 55 70 56 56 48 51 51 57 53 54 70 57 52 48 53 49 56 57 50 66 70 51 53], Version : 193509558, meta: 2, userMeta: 0 valuePointer: {Fid:254 Len:26293 Offset:363430027}
Jul 15 11:33:17 celestia[2012453]: 2024-07-15T11:33:17.016Z INFO header/store store/store.go:367 new head {"height": 2268972, "hash": "8C8AAA076A5861D7A308A9031717E2AD5488E1AA1A46F01BF5FD19AA543FD04D"}
Jul 15 11:33:27 celestia[2012453]: 2024-07-15T11:33:27.655Z WARN badger4 [email protected]/db.go:460 Block cache might be too small. Metrics: hit: 189025 miss: 46988745 keys-added: 118261 keys-updated: 1332 keys-evicted: 118235 cost-added: 588947648985 cost-evicted: 588688198156 sets-dropped: 1408 sets-rejected: 2230925 gets-dropped: 34126464 gets-kept: 13048640 gets-total: 47177770 hit-ratio: 0.00
Jul 15 11:33:27 celestia[2012453]: 2024-07-15T11:33:27.655Z WARN badger4 [email protected]/db.go:461 Cache life expectancy (in seconds):
Jul 15 11:33:27 celestia[2012453]: -- Histogram:
Jul 15 11:33:27 celestia[2012453]: Min value: 0
Jul 15 11:33:27 celestia[2012453]: Max value: 2558
Jul 15 11:33:27 celestia[2012453]: Count: 117531
Jul 15 11:33:27 celestia[2012453]: 50p: 2.00
Jul 15 11:33:27 celestia[2012453]: 75p: 2.00
Jul 15 11:33:27 celestia[2012453]: 90p: 8.00
Jul 15 11:33:27 celestia[2012453]: [0, 2) 101104 86.02% 86.02%
Jul 15 11:33:27 celestia[2012453]: [2, 4) 3060 2.60% 88.63%
Jul 15 11:33:27 celestia[2012453]: [4, 8) 3248 2.76% 91.39%
Jul 15 11:33:27 celestia[2012453]: [8, 16) 3550 3.02% 94.41%
Jul 15 11:33:27 celestia[2012453]: [16, 32) 2641 2.25% 96.66%
Jul 15 11:33:27 celestia[2012453]: [32, 64) 1903 1.62% 98.28%
Jul 15 11:33:27 celestia[2012453]: [64, 128) 1081 0.92% 99.20%
Jul 15 11:33:27 celestia[2012453]: [128, 256) 574 0.49% 99.69%
Jul 15 11:33:27 celestia[2012453]: [256, 512) 294 0.25% 99.94%
Jul 15 11:33:27 celestia[2012453]: [512, 1024) 69 0.06% 99.99%
Jul 15 11:33:27 celestia[2012453]: [1024, 2048) 6 0.01% 100.00%
Jul 15 11:33:27 celestia[2012453]: [2048, 4096) 1 0.00% 100.00%
Jul 15 11:33:27 celestia[2012453]: --
Jul 15 11:33:29 celestia[2012453]: 2024-07-15T11:33:29.795Z INFO header/store store/store.go:367 new head {"height": 2268973, "hash": "651E0ECA01C040C47266BC1F944D80FDCD976C812575996E383D93D97CE9FBFA"}
Jul 15 11:33:41 celestia[2012453]: 2024-07-15T11:33:41.282Z INFO header/store store/store.go:367 new head {"height": 2268974, "hash": "B9E3EFE6F3380C0B7C902D942BC3C2EA4D84C64843098ED5DC343E5F1BCF632B"}
Jul 15 11:33:53 celestia[2012453]: 2024-07-15T11:33:53.078Z INFO header/store store/store.go:367 new head {"height": 2268975, "hash": "1AFA9682C8F0671DA80931ACB938E24E96575C395F4D00A788645A34ACF77CB5"}
Jul 15 11:34:04 celestia[2012453]: 2024-07-15T11:34:04.937Z INFO header/store store/store.go:367 new head {"height": 2268976, "hash": "9D7D6350A3D43F5BE1146DC352D2BE6F1C1FA0B4DCA5F574BC32DBFEDE343F68"}
Jul 15 11:34:16 celestia[2012453]: 2024-07-15T11:34:16.719Z INFO header/store store/store.go:367 new head {"height": 2268977, "hash": "009612AFF66C96D5E4CEEB0FBB855C8F2A72A7FA85160101803C7AE5D97D4616"}
Badger might not stop immediately, if compaction is in process. Sigkill might result in data corruption during compaction. Right now compaction might take few minutes, because of large inverted_index. The index will be removed after shwap, so it should not be a problem.
Badger might not stop immediately, if compaction is in process. Sigkill might result in data corruption during compaction. Right now compaction might take few minutes, because of large inverted_index. The index will be removed after shwap, so it should not be a problem.
I don't think this is the case. As you can see (discord screenshots) there was no intensive activity on the disk after the command and a classic reboot doesn't cause this
I just shut down to decommit the bridge node and the same problem occurred again. I hope I have been helpful in these months of testnet. I will no longer be able to follow the issues
CPU usage spike during a shutdown (v0.20.4 mainnet). i've experienced this issue since as long as I can remember (before I reported it in March)
