lightchain icon indicating copy to clipboard operation
lightchain copied to clipboard

Published 20 hours ago • verified publisher icon lightstreams-network

Fix LevelDB panic when shutting down the node

Open EnchanterIO opened this issue 5 years ago • 2 comments

How to reproduce

  • Boot up a new no-synced node
  • Before node is fully sync force a shut down (CTRL+C)
Screen Shot 2020-02-11 at 13 28 50

Steps to resolve:

  1. Shutting down the node should close all goroutines gracefully

EnchanterIO avatar Feb 11 '20 12:02 EnchanterIO

Current status After the first research over this issue I could narrow the issue to the scenario where a node being shut down triggers tendermint dbs to close down and it appears to happen that the tendermint stateDb is closed and Tendermint node try to persist a block right after that happens. This is a racing condition and it does not happen on every shut down.

Next steps Try to validate the potential solution of stopping Tendermint ABCI reactor before DBs are closed

ggarri avatar Mar 17 '20 18:03 ggarri

Seems like the 1s timeout should be enough for persisting the changes to disk. If it fails we increase it or implement a proper solution. I am bumping the version and will test it for few days on my node. If everything goes fine we merge it by the end of the month 👍

EnchanterIO avatar Mar 24 '20 13:03 EnchanterIO