neo-go
neo-go copied to clipboard
nspcc-dev
T5 node can't init MPT after OOM killer at height 1779831
The node was killed by OOM killer, and after that an automatic node restart fails due to missing stateroot:
Aug 3 13:54:46 tenes4 neogo-rpc[66705]: 2023-08-03T13:54:46.796Z#011INFO#011started protocol#011{"addr": "170.64.152.72:49840", "userAgent": "/NEO-GO:0.101.4/", "startHeight": 1829186, "id": 3239957722}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.083Z#011WARN#011peer disconnected#011{"addr": "20.55.44.83:20333", "error": "ping/pong timeout", "peerCount": 25}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.086Z#011WARN#011peer disconnected#011{"addr": "104.43.9.164:20333", "error": "ping/pong timeout", "peerCount": 24}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.087Z#011WARN#011peer disconnected#011{"addr": "20.127.137.131:20333", "error": "ping/pong timeout", "peerCount": 23}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.087Z#011WARN#011peer disconnected#011{"addr": "157.230.27.189:20335", "error": "ping/pong timeout", "peerCount": 22}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.088Z#011WARN#011peer disconnected#011{"addr": "13.68.253.120:20333", "error": "ping/pong timeout", "peerCount": 21}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.088Z#011WARN#011peer disconnected#011{"addr": "20.228.234.187:20333", "error": "ping/pong timeout", "peerCount": 20}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.089Z#011WARN#011peer disconnected#011{"addr": "20.127.87.89:20333", "error": "ping/pong timeout", "peerCount": 19}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.089Z#011WARN#011peer disconnected#011{"addr": "104.215.144.118:20333", "error": "ping/pong timeout", "peerCount": 18}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.089Z#011WARN#011peer disconnected#011{"addr": "13.68.251.200:20333", "error": "ping/pong timeout", "peerCount": 17}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.089Z#011WARN#011peer disconnected#011{"addr": "157.90.177.38:30333", "error": "ping/pong timeout", "peerCount": 16}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.089Z#011WARN#011peer disconnected#011{"addr": "65.108.97.115:41220", "error": "ping/pong timeout", "peerCount": 15}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "13.67.116.69:20333", "error": "ping/pong timeout", "peerCount": 14}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "20.84.53.54:20333", "error": "ping/pong timeout", "peerCount": 13}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "20.127.85.47:20333", "error": "ping/pong timeout", "peerCount": 12}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "167.71.212.120:21333", "error": "ping/pong timeout", "peerCount": 11}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "20.198.226.132:20333", "error": "ping/pong timeout", "peerCount": 10}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "104.215.158.208:20333", "error": "ping/pong timeout", "peerCount": 9}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "13.214.218.27:20333", "error": "ping/pong timeout", "peerCount": 8}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "85.159.213.129:20333", "error": "ping/pong timeout", "peerCount": 7}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.090Z#011WARN#011peer disconnected#011{"addr": "104.43.11.229:20333", "error": "ping/pong timeout", "peerCount": 6}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.321Z#011INFO#011new peer connected#011{"addr": "13.68.251.200:20333", "peerCount": 7}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.322Z#011INFO#011new peer connected#011{"addr": "20.84.53.54:20333", "peerCount": 8}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.402Z#011INFO#011started protocol#011{"addr": "13.68.251.200:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1971924759}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.402Z#011INFO#011started protocol#011{"addr": "20.84.53.54:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1986176990}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.419Z#011INFO#011new peer connected#011{"addr": "157.90.177.38:30333", "peerCount": 9}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.427Z#011INFO#011new peer connected#011{"addr": "68.183.211.18:10333", "peerCount": 10}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.480Z#011INFO#011new peer connected#011{"addr": "20.55.44.83:20333", "peerCount": 11}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.481Z#011INFO#011new peer connected#011{"addr": "88.198.46.21:20333", "peerCount": 12}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.487Z#011INFO#011new peer connected#011{"addr": "104.43.9.164:20333", "peerCount": 13}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.531Z#011INFO#011new peer connected#011{"addr": "20.25.114.208:20333", "peerCount": 14}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.531Z#011INFO#011new peer connected#011{"addr": "20.127.137.131:20333", "peerCount": 15}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.531Z#011INFO#011new peer connected#011{"addr": "20.228.234.187:20333", "peerCount": 16}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.557Z#011INFO#011started protocol#011{"addr": "20.55.44.83:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1424413026}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.602Z#011INFO#011started protocol#011{"addr": "68.183.211.18:10333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1798089428}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.603Z#011INFO#011started protocol#011{"addr": "157.90.177.38:30333", "userAgent": "/NEO-GO:0.101.4/", "startHeight": 2458335, "id": 1649111613}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.611Z#011INFO#011started protocol#011{"addr": "20.25.114.208:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 639618423}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.611Z#011INFO#011started protocol#011{"addr": "20.127.137.131:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 523520231}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.611Z#011INFO#011started protocol#011{"addr": "20.228.234.187:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1255773570}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.646Z#011INFO#011started protocol#011{"addr": "104.43.9.164:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 717865152}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.647Z#011INFO#011new peer connected#011{"addr": "104.215.158.208:20333", "peerCount": 17}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.659Z#011INFO#011started protocol#011{"addr": "88.198.46.21:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 912968498}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.666Z#011INFO#011new peer connected#011{"addr": "13.214.218.27:20333", "peerCount": 18}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.697Z#011INFO#011new peer connected#011{"addr": "13.68.253.120:20333", "peerCount": 19}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.773Z#011INFO#011new peer connected#011{"addr": "85.159.213.129:20333", "peerCount": 20}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.775Z#011INFO#011new peer connected#011{"addr": "157.230.27.189:20335", "peerCount": 21}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.778Z#011INFO#011started protocol#011{"addr": "13.68.253.120:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1244792231}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.803Z#011INFO#011started protocol#011{"addr": "104.215.158.208:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 44922377}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.870Z#011INFO#011started protocol#011{"addr": "13.214.218.27:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 368113034}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.879Z#011INFO#011new peer connected#011{"addr": "167.71.212.120:37726", "peerCount": 22}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.896Z#011INFO#011new peer connected#011{"addr": "104.43.11.229:20333", "peerCount": 23}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.929Z#011INFO#011new peer connected#011{"addr": "157.90.177.38:20333", "peerCount": 24}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.930Z#011INFO#011started protocol#011{"addr": "85.159.213.129:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1870979988}
Aug 3 13:56:05 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:05.979Z#011INFO#011new peer connected#011{"addr": "20.55.43.229:20333", "peerCount": 25}
Aug 3 13:56:06 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:06.074Z#011INFO#011started protocol#011{"addr": "104.43.11.229:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 1294269051}
Aug 3 13:56:06 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:06.076Z#011INFO#011started protocol#011{"addr": "167.71.212.120:37726", "userAgent": "/NEO-GO:0.101.4/", "startHeight": 1805891, "id": 251387157}
Aug 3 13:56:06 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:06.076Z#011INFO#011started protocol#011{"addr": "20.55.43.229:20333", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 2037889247}
Aug 3 13:56:06 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:06.107Z#011INFO#011started protocol#011{"addr": "157.90.177.38:20333", "userAgent": "/NEO-GO:0.101.4/", "startHeight": 2458335, "id": 678207779}
Aug 3 13:56:06 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:06.545Z#011INFO#011started protocol#011{"addr": "157.230.27.189:20335", "userAgent": "/Neo:3.5.0/", "startHeight": 2458335, "id": 659655972}
Aug 3 13:56:19 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:19.589Z#011INFO#011new peer connected#011{"addr": "65.108.97.115:45262", "peerCount": 26}
Aug 3 13:56:19 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:19.767Z#011INFO#011started protocol#011{"addr": "65.108.97.115:45262", "userAgent": "/Neo:3.5.0/", "startHeight": 2458336, "id": 433871748}
Aug 3 13:56:35 tenes4 neogo-rpc[66705]: 2023-08-03T13:56:35.078Z#011WARN#011peer disconnected#011{"addr": "170.64.152.72:49840", "error": "ping/pong timeout", "peerCount": 25}
Aug 3 13:57:06 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:06.362Z#011INFO#011persisted to disk#011{"blocks": 23250, "keys": 607349, "headerHeight": 1779687, "blockHeight": 1779686, "took": "33m22.925018945s"}
Aug 3 13:57:09 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:09.473Z#011INFO#011new peer connected#011{"addr": "170.64.152.72:54398", "peerCount": 26}
Aug 3 13:57:09 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:09.707Z#011INFO#011started protocol#011{"addr": "170.64.152.72:54398", "userAgent": "/NEO-GO:0.101.4/", "startHeight": 1831087, "id": 3239957722}
Aug 3 13:57:09 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:09.847Z#011INFO#011persisted to disk#011{"blocks": 1, "keys": 223, "headerHeight": 1779687, "blockHeight": 1779687, "took": "2.968632296s"}
Aug 3 13:57:12 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:12.975Z#011INFO#011persisted to disk#011{"blocks": 6, "keys": 119, "headerHeight": 1779694, "blockHeight": 1779693, "took": "2.040715346s"}
Aug 3 13:57:13 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:13.812Z#011INFO#011persisted to disk#011{"blocks": 8, "keys": 153, "headerHeight": 1779702, "blockHeight": 1779701, "took": "835.526817ms"}
Aug 3 13:57:20 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:20.143Z#011INFO#011persisted to disk#011{"blocks": 6, "keys": 117, "headerHeight": 1779708, "blockHeight": 1779707, "took": "5.131703632s"}
Aug 3 13:57:21 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:21.709Z#011INFO#011persisted to disk#011{"blocks": 5, "keys": 300, "headerHeight": 1779712, "blockHeight": 1779712, "took": "1.565643022s"}
Aug 3 13:57:22 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:22.147Z#011INFO#011persisted to disk#011{"blocks": 1, "keys": 23, "headerHeight": 1779713, "blockHeight": 1779713, "took": "437.183218ms"}
Aug 3 13:57:24 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:24.281Z#011INFO#011persisted to disk#011{"blocks": 7, "keys": 134, "headerHeight": 1779721, "blockHeight": 1779720, "took": "1.564692499s"}
Aug 3 13:57:25 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:25.624Z#011INFO#011persisted to disk#011{"blocks": 7, "keys": 135, "headerHeight": 1779727, "blockHeight": 1779727, "took": "1.342682214s"}
Aug 3 13:57:29 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:29.469Z#011INFO#011persisted to disk#011{"blocks": 12, "keys": 410, "headerHeight": 1779739, "blockHeight": 1779739, "took": "3.845028977s"}
Aug 3 13:57:31 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:31.902Z#011INFO#011new peer connected#011{"addr": "167.71.212.120:47418", "peerCount": 27}
Aug 3 13:57:31 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:31.904Z#011WARN#011peer disconnected#011{"addr": "167.71.212.120:37726", "error": "write tcp 5.78.81.79:21333->167.71.212.120:37726: write: connection reset by peer", "peerCount": 26}
Aug 3 13:57:32 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:32.094Z#011INFO#011started protocol#011{"addr": "167.71.212.120:47418", "userAgent": "/NEO-GO:0.101.4/", "startHeight": 1806993, "id": 251387157}
Aug 3 13:57:37 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:37.720Z#011INFO#011persisted to disk#011{"blocks": 73, "keys": 1834, "headerHeight": 1779812, "blockHeight": 1779812, "took": "8.249994063s"}
Aug 3 13:57:37 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:37.799Z#011INFO#011persisted to disk#011{"blocks": 0, "keys": 2, "headerHeight": 1779813, "blockHeight": 1779812, "took": "76.834588ms"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.028Z#011WARN#011contract invocation failed#011{"tx": "1cc31ce3f7626517e69a9c9238660b9883b90212747306e7644a06b130ef3d0f", "block": 1780009, "error": "at instruction 15 (SYSCALL): insufficient gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.088Z#011WARN#011contract invocation failed#011{"tx": "6a5feb3ff5f20d3e56aeb29499b0af32a05a9df1368e0c1f981cd537a69d098c", "block": 1780011, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.089Z#011WARN#011contract invocation failed#011{"tx": "f5ed956e695b21ab8216e5fbb6d4afb99240a8378a7ecba74ddff5d0f338c132", "block": 1780011, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.176Z#011WARN#011contract invocation failed#011{"tx": "cacdbb42f2971bc00003d7dc9e978c5c9443577ad69ac092c004a36fb4cb51a0", "block": 1780019, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.176Z#011WARN#011contract invocation failed#011{"tx": "deeb8a1ff787ed52e71cc79195229ad1e9694b4e3c132624b7141956e72d7f06", "block": 1780019, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.291Z#011WARN#011contract invocation failed#011{"tx": "a677280e467a4a7172e3bf406680fe0102e3cef61acab048cff46dff359588c0", "block": 1780037, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.319Z#011WARN#011contract invocation failed#011{"tx": "57827912530cb2bbaad573fdbab15611f54579698692f41f8cea42a042299c62", "block": 1780043, "error": "at instruction 15 (SYSCALL): insufficient gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.364Z#011WARN#011contract invocation failed#011{"tx": "7e4eee354de629611f5982f7d93cbd4c6ccb34baab87021773183bceee2f5de4", "block": 1780045, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.365Z#011WARN#011contract invocation failed#011{"tx": "867b1a15035bba545e646253b3d4342c1bf292f30f2293cdd9ab1f5bf10a7509", "block": 1780045, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.411Z#011WARN#011contract invocation failed#011{"tx": "30f798a2472422c053672a4ff866477008d218869ad703aedbc5d8a6ca52255a", "block": 1780054, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.411Z#011WARN#011contract invocation failed#011{"tx": "673525239a5d7a2f85e16b4902a373e9918d5c35abfb1b6a2a747162aa7ba51d", "block": 1780054, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.580Z#011WARN#011contract invocation failed#011{"tx": "b9b91368f0318a0ce1ef51372e401b0948967331f583181ff6a5ba3916820c22", "block": 1780099, "error": "at instruction 80 (SYSCALL): [System.Storage](http://system.storage/).Get failed: insufficient amount of gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.604Z#011WARN#011contract invocation failed#011{"tx": "cea7e675d18a97d8163d10a921f657e901bce2fa4e09b8f5a0529ee6a0f87893", "block": 1780102, "error": "at instruction 80 (SYSCALL): [System.Storage](http://system.storage/).Get failed: insufficient amount of gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.644Z#011WARN#011contract invocation failed#011{"tx": "cbc4aaf361777e0779aa99176c979a76c13f5a9487a20e6eea5c5dd5fc6960f7", "block": 1780105, "error": "at instruction 80 (SYSCALL): [System.Storage](http://system.storage/).Get failed: insufficient amount of gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.651Z#011WARN#011contract invocation failed#011{"tx": "4bcc8b6a474fd44afa9fd114be974e3cab47b834a094dacc5fda81da1b1c3082", "block": 1780106, "error": "at instruction 15 (SYSCALL): insufficient gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.674Z#011WARN#011contract invocation failed#011{"tx": "ebea67ce513de7538e4e6750c5eb01c4d6dc2fc9618c1bcfc250e26f40e55cd9", "block": 1780110, "error": "at instruction 15 (SYSCALL): insufficient gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.685Z#011WARN#011contract invocation failed#011{"tx": "7bc63aeaba75a43040370986ef3a037587d78724c94e30642782d1f7756dade5", "block": 1780112, "error": "at instruction 15 (SYSCALL): insufficient gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.697Z#011WARN#011contract invocation failed#011{"tx": "0a3ee8c55fc5c346d742b1cb0b1289d783bda59517f01f0e60aa35dbfd3782e9", "block": 1780115, "error": "at instruction 15 (SYSCALL): insufficient gas"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.760Z#011WARN#011contract invocation failed#011{"tx": "b819a3b0c154ff74a0731493e4195d595acab87fa486e28a5564e437c1b234f6", "block": 1780122, "error": "at instruction 1 (SYSCALL): failed native call: at instruction 78 (SYSCALL): EOF"}
Aug 3 13:57:41 tenes4 neogo-rpc[66705]: 2023-08-03T13:57:41.769Z#011INFO#011persisted to disk#011{"blocks": 19, "keys": 545, "headerHeight": 1779831, "blockHeight": 1779831, "took": "3.046210121s"}
Aug 3 13:58:09 tenes4 neogo-rpc[69514]: 2023-08-03T13:58:09.974Z#011INFO#011MaxValidUntilBlockIncrement is not set or wrong, using default value#011{"MaxValidUntilBlockIncrement": 5760}
Aug 3 13:58:09 tenes4 neogo-rpc[69514]: 2023-08-03T13:58:09.982Z#011INFO#011restoring blockchain#011{"version": "0.2.8"}
Aug 3 13:58:13 tenes4 neogo-rpc[69514]: could not initialize blockchain: can't init MPT at height 1779831: key not found
This issue is likely to be caused by some bolt persisting problem. It's clear from the application side, we have an atomic bolt transaction that contains all block-related changes (including stateroot) and either should be persisted or should be rejected.
tenes1 has successfully synced mainchain, but failed on sidechain:
Aug 07 10:30:27 tenes1 systemd[1]: neofs-ir.service: Main process exited, code=exited, status=1/FAILURE
Aug 07 10:30:27 tenes1 systemd[1]: neofs-ir.service: Failed with result 'exit-code'.
Aug 07 10:30:32 tenes1 systemd[1]: neofs-ir.service: Scheduled restart job, restart counter is at 19016.
Aug 07 10:30:32 tenes1 systemd[1]: Stopped NeoFS InnerRing node.
Aug 07 10:30:32 tenes1 systemd[1]: Started NeoFS InnerRing node.
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:229 initial gas supply is not set or wrong, setting default value {"InitialGASSupply": "52000000"}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:233 mempool size is not set or wrong, setting default value {"MemPoolSize": 50000}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:237 P2PNotaryRequestPayloadPool size is not set or wrong, setting default value {"P2PNotaryRequestPayloadPoolSize": 1000}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:241 MaxBlockSize is not set or wrong, setting default value {"MaxBlockSize": 262144}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:245 MaxBlockSystemFee is not set or wrong, setting default value {"MaxBlockSystemFee": 900000000000}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:253 MaxTransactionsPerBlock is not set or wrong, using default value {"MaxTransactionsPerBlock": 512}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:271 MaxValidUntilBlockIncrement is not set or wrong, using default value {"MaxValidUntilBlockIncrement": 5760}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:286 NativeActivations are not set, using default values
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.395Z info core/blockchain.go:290 Hardforks are not set, using default value
Aug 07 10:30:33 tenes1 neofs-ir[329555]: 2023-08-07T10:30:33.398Z info core/blockchain.go:429 restoring blockchain {"version": "0.2.8"}
Aug 07 10:30:33 tenes1 neofs-ir[329555]: init internal blockchain: init core blockchain component: can't init MPT at height 2043034: key not found
Aug 07 10:30:33 tenes1 systemd[1]: neofs-ir.service: Main process exited, code=exited, status=1/FAILURE
Aug 07 10:30:33 tenes1 systemd[1]: neofs-ir.service: Failed with result 'exit-code'.
Aug 07 10:30:38 tenes1 systemd[1]: neofs-ir.service: Scheduled restart job, restart counter is at 19017.
Aug 07 10:30:38 tenes1 systemd[1]: Stopped NeoFS InnerRing node.
Aug 07 10:30:38 tenes1 systemd[1]: Started NeoFS InnerRing node.
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:229 initial gas supply is not set or wrong, setting default value {"InitialGASSupply": "52000000"}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:233 mempool size is not set or wrong, setting default value {"MemPoolSize": 50000}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:237 P2PNotaryRequestPayloadPool size is not set or wrong, setting default value {"P2PNotaryRequestPayloadPoolSize": 1000}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:241 MaxBlockSize is not set or wrong, setting default value {"MaxBlockSize": 262144}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:245 MaxBlockSystemFee is not set or wrong, setting default value {"MaxBlockSystemFee": 900000000000}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:253 MaxTransactionsPerBlock is not set or wrong, using default value {"MaxTransactionsPerBlock": 512}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:271 MaxValidUntilBlockIncrement is not set or wrong, using default value {"MaxValidUntilBlockIncrement": 5760}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:286 NativeActivations are not set, using default values
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.469Z info core/blockchain.go:290 Hardforks are not set, using default value
Aug 07 10:30:39 tenes1 neofs-ir[329563]: 2023-08-07T10:30:39.473Z info core/blockchain.go:429 restoring blockchain {"version": "0.2.8"}
Aug 07 10:30:39 tenes1 neofs-ir[329563]: init internal blockchain: init core blockchain component: can't init MPT at height 2043034: key not found
Aug 07 10:30:39 tenes1 systemd[1]: neofs-ir.service: Main process exited, code=exited, status=1/FAILURE
Aug 07 10:30:39 tenes1 systemd[1]: neofs-ir.service: Failed with result 'exit-code'.
