ceremonyclient
ceremonyclient copied to clipboard
QuilibriumNetwork
last version crash, panic: get last data clock frame: get data clock frame: pebble: not found
other node:
{"level":"info","ts":1711282886.636553,"caller":"ceremony/ceremony_execution_engine.go:347","msg":"adding retroactive peer reward info"}
node-1 | {"level":"info","ts":1711282887.8062675,"caller":"ceremony/ceremony_execution_engine.go:406","msg":"proving execution output for inclusion"}
node-1 | {"level":"info","ts":1711282888.1587415,"caller":"ceremony/ceremony_execution_engine.go:415","msg":"creating kzg proof"}
node-1 | {"level":"info","ts":1711282888.5081005,"caller":"ceremony/ceremony_execution_engine.go:425","msg":"finalizing execution proof"}
node-1 | {"level":"info","ts":1711282888.5084324,"caller":"ceremony/ceremony_data_clock_consensus_engine.go:248","msg":"constructing consensus engine"}
node-1 | {"level":"info","ts":1711282888.5106964,"caller":"master/master_clock_consensus_engine.go:136","msg":"constructing consensus engine"}
node-1 | {"level":"info","ts":1711282888.510769,"caller":"app/node.go:49","msg":"running compaction"}
node-1 | panic: compact: pebble: not found
node-1 |
node-1 | goroutine 1 [running]:
node-1 | source.quilibrium.com/quilibrium/monorepo/node/app.newNode(0xc0000403f0, {0x1dfb2a0?, 0xc0028a8588}, {0x1df1370?, 0xc001e8e500}, {0x1dfa4f0?, 0xc0002f4ee0}, 0xc0046ed400, {0x1df1318, 0xc002ea2840})
node-1 | /opt/ceremonyclient/node/app/node.go:54 +0x3c6
node-1 | source.quilibrium.com/quilibrium/monorepo/node/app.NewNode(0xc00008af00, 0x7?)
node-1 | /opt/ceremonyclient/node/app/wire_gen.go:43 +0x4a7
node-1 | main.main()
node-1 | /opt/ceremonyclient/node/main.go:182 +0x4ee
restart cycle，other node
node-1 | {"level":"info","ts":1711344333.7467315,"caller":"ceremony/ceremony_execution_engine.go:434","msg":"ceremony data loaded","g2_power":"AwI3CiNfZFrFVJiMleegL9vxKKCuIlC39r0TThV4UvT6tlGd/eDHlpYrS8NS8ecK5YXc1qQjTz31E9PsZE5LRTKdKKdWC6SitKoCyduvXYhPEXqmZujPjCiZY0yqHDFMzvFHRUDORHbH9r72c7NIYnctV8ZQpO1YX771TlB1F0v4tOSqZ85MXxX1STIPTrv34A44Dr2I1TAQK67KoESqW81ccOEm4Cb78hnIKzCRgLlTH38w5GG8hFMoT38xTvJJ2xcYNSa6yxYmEEm9OorQMBjYL/QZAMo1dV0BoQ7oMyivFDrkZIVTXVtjcLbuRIJxHl0F03zlNwVfWjHERPUrCoHflbMD/R3r/0Y1dzmbcKsql73YlADIcs4oO8jAsqW2To0PqJgJanVE2RKM++5zl91X5PQ244kRwkY/WZG+NBU41pQx1Dn7Rup/kRc16Ufllww5QFOQs4DOu0D6vKKZSr9wX8avG1pzo2z+JG+YB8M8/uRXa3MEM1QeUzBFGdI9jWbq0PJqoAzbnYFsu+bwhNbsZf7jeyURKgs6Z5SByDjC0tGNjV/MapPUSQ/QuAKtguaO9ektbASv2XJP95e7iqaje3sE7Q5pKv9NJPuOmT5VR2ONM89kRmlapr8sD4hVS3BS6qbgYzQ7WFtFNkJtzgS4Y00Pya0OSyNRxZbwTawMqUuu4LHQajpwPkOD9j7ndJPF4Jbl2Is7oJSi95DMSYG07mXFng1xwDXDicspef0HjRRzKYJtCeSEq/WmjC4hIP+K0SzMOpju"}
node-1 | {"level":"info","ts":1711344333.7467532,"caller":"ceremony/ceremony_data_clock_consensus_engine.go:265","msg":"starting ceremony consensus engine"}
node-1 | {"level":"info","ts":1711344333.7467594,"caller":"ceremony/ceremony_data_clock_consensus_engine.go:270","msg":"loading last seen state"}
node-1 | {"level":"info","ts":1711344333.7468684,"caller":"master/master_clock_consensus_engine.go:202","msg":"peers in store","peer_store_count":207,"network_peer_count":48}
node-1 | panic: get latest data clock frame: get data clock frame: pebble: not found
node-1 |
node-1 | goroutine 363783 [running]:
node-1 | source.quilibrium.com/quilibrium/monorepo/node/consensus/time.(*DataTimeReel).Start(0xc003e5cd80)
node-1 | /opt/ceremonyclient/node/consensus/time/data_time_reel.go:126 +0x2fd
node-1 | source.quilibrium.com/quilibrium/monorepo/node/consensus/ceremony.(*CeremonyDataClockConsensusEngine).Start(0xc003c9d440)
node-1 | /opt/ceremonyclient/node/consensus/ceremony/ceremony_data_clock_consensus_engine.go:271 +0x96
node-1 | source.quilibrium.com/quilibrium/monorepo/node/execution/intrinsics/ceremony.(*CeremonyExecutionEngine).Start.func1()
node-1 | /opt/ceremonyclient/node/execution/intrinsics/ceremony/ceremony_execution_engine.go:440 +0x33
node-1 | created by source.quilibrium.com/quilibrium/monorepo/node/execution/intrinsics/ceremony.(*CeremonyExecutionEngine).Start
node-1 | /opt/ceremonyclient/node/execution/intrinsics/ceremony/ceremony_execution_engine.go:439 +0x198
node-1 | %#########
node-1 | #############################
node-1 | ########################################&
node-1 | ###############################################
node-1 | &#####################% %######################
node-1 | ################# #################
node-1 | ############### ###############
node-1 | ############# ##############
node-1 | ############# ############&
node-1 | ############ ############
node-1 | ########### ########## &###########
node-1 | ########### ############## ###########
node-1 | ########### ############## ##########&
node-1 | ########## ############## ##########
node-1 | %########## ########## ##########
node-1 | ########## ##########
node-1 | ########## &#########
node-1 | ########## ####### ####### ##########
node-1 | %######### &######################### ##########
node-1 | ########## ##############% ############## &##########
node-1 | ' ' &############## ############### ##########
node-1 | ' ' ############### ##############% ###########
node-1 | ' '. ########## ############### ########
node-1 | '. . ##### ##############% ####
node-1 | ' '. ###############
node-1 | '. '.. ##############%
node-1 | '. '-. ###############
node-1 | '-. ''-.. .. ##############%
node-1 | '-. ''---............----' '. ###############
node-1 | '-.. '. ############
node-1 | ''-.. ..' ########
node-1 | ''---.. ...---'' ##
node-1 | ''----------''
node-1 |
node-1 | Quilibrium Node - v1.4.12 – Sunset
node-1 |
node-1 | Loading ceremony state and starting node...
node-1 | 2024/03/25 05:26:43 [JOB 1] WAL file .config/store/000224.log with log number 000224 stopped reading at offset: 0; replayed 0 keys in 0 batches
node-1 | 2024/03/25 05:26:43 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
node-1 | {"level":"info","ts":1711344403.5423057,"caller":"p2p/blossomsub.go:139","msg":"established peer id","peer_id":"QmdQqnTwyq9hvDJqfTbpR68PApqeKcQML8mKWLWb62aYHp"}
node-1 | {"level":"info","ts":1711344403.5423286,"caller":"p2p/blossomsub.go:332","msg":"establishing dht"}
node-1 | {"level":"info","ts":1711344403.5424187,"caller":"p2p/blossomsub.go:350","msg":"connecting to bootstrap","peer_id":"QmdQqnTwyq9hvDJqfTbpR68PApqeKcQML8mKWLWb62aYHp"}
node-1 | {"level":"info","ts":1711344403.5662725,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmafiAXLu1JWktyfzDtD67i78GRBYCfQ4doTfq7pp7wfQ1"}
node-1 | {"level":"info","ts":1711344403.6005123,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmaVaJ93mKaaqDnsYbHGEUBzEtbWhb3s4DAutt6JeJC5Wz"}
node-1 | {"level":"info","ts":1711344403.6021044,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmP8C7g9ZRiWzhqN2AgFu5onS6HwHzR6Vv1TCHxAhnCSnq"}
node-1 | {"level":"info","ts":1711344403.6099212,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"Qmc35n99eojSvW3PkbfBczJoSX92WmnnKh3Fg114ok3oo4"}
node-1 | {"level":"info","ts":1711344403.7168107,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"Qmekz5obb9qCRP5CrZ4D8Tmabbr5mJf6mgBJHTaitrx7Fx"}
node-1 | {"level":"info","ts":1711344403.735128,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmeqBjm3iX7sdTieyto1gys5ruQrQNPKfaTGcVQQWJPYDV"}
node-1 | {"level":"info","ts":1711344403.7695372,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmcKQjpQmLpbDsiif2MuakhHFyxWvqYauPsJDaXnLav7PJ"}
node-1 | {"level":"info","ts":1711344403.791284,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmR7jT9NKL3yZ1iRjLxPJjAVNbk38HAfDtAKF5SCZmHdpK"}
node-1 | {"level":"info","ts":1711344403.8641841,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmUSbMytVBUYiiGE266aZHrHrP17vLx5UJFd7o74HkDoaV"}
node-1 | {"level":"info","ts":1711344403.882016,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"Qme2LDknxE5z8AnECTjJ6ZmTkSYmoQ8phegaD5vEcZM9uB"}
node-1 | {"level":"info","ts":1711344408.5435307,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmW6QDvKuYqJYYMP5tMZSp12X3nexywK28tZNgqtqNpEDL:\n * [/ip4/186.233.184.181/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.5435777,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmZejZ8DBGQ6foX9recW73GA6TqL6hCMX9ETWWW1Fb8xtx:\n * [/ip4/144.76.104.93/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.543669,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmYVaHXdFmHFeTa6oPixgjMVag6Ex7gLjE559ejJddwqzu:\n * [/ip4/51.15.18.247/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.543669,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmPBYgDy7snHon7PAn8nv1shApQBQz1iHb2sBBS8QSgQwW:\n * [/ip4/207.246.81.38/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.5513425,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmUhm9iZVruSxyavjoPLCfuoRG94SGQEkfxEEoukEZmD5B:\n * [/ip4/91.242.214.79/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.5572686,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmXaSyf9Gaq5ddcHysefDpuE7SJqzdg1surQ5wac55ysJV:\n * [/ip4/185.209.179.129/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.5593953,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmXbbmtS5D12rEc4HWiHWr6e83SCE4jeThPP4VJpAQPvXq:\n * [/ip4/97.118.73.17/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.5942452,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmS7C1UhN8nvzLJgFFf1uspMRrXjJqThHNN6AyEXp6oVUB:\n * [/ip6/2001:41d0:8:823b::/udp/8336/quic] INTERNAL_ERROR (local): write udp6 [::]:33728->[2001:41d0:8:823b::]:8336: sendmsg: network is unreachable\n * [/ip4/5.39.66.59/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344408.5954552,"caller":"p2p/blossomsub.go:371","msg":"error while connecting to dht peer","error":"failed to dial: failed to dial QmQaFmbYVrKSwoen5UQdaqyDq4QhXfSSLDVnYpYD4SF9tX:\n * [/ip4/87.212.22.230/udp/8336/quic] timeout: no recent network activity"}
node-1 | {"level":"info","ts":1711344463.6394405,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"Qmd233pLUDvcDW3ama27usfbG1HxKNh1V9dmWVW1SXp1pd"}
node-1 | {"level":"info","ts":1711344463.6602273,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmQkyqziNCycUDdeBypkikMZWH2bBmmxfK7cbJFJmyyQdQ"}
node-1 | {"level":"info","ts":1711344465.3761308,"caller":"p2p/blossomsub.go:373","msg":"connected to peer","peer_id":"QmQn3bWk5aqaNSv9dwPjBg4qdeGBGNEB72tvuhgEc64Ki5"}
node-1 | {"level":"info","ts":1711344465.3762128,"caller":"p2p/blossomsub.go:553","msg":"initiating peer discovery"}
node-1 | {"level":"info","ts":1711344831.531603,"caller":"p2p/blossomsub.go:594","msg":"completed initial peer discovery"}
node-1 | {"level":"info","ts":1711344831.5496242,"caller":"ceremony/ceremony_data_clock_consensus_engine.go:248","msg":"constructing consensus engine"}
node-1 | {"level":"info","ts":1711344831.550557,"caller":"master/master_clock_consensus_engine.go:136","msg":"constructing consensus engine"}
node-1 | {"level":"info","ts":1711344831.550606,"caller":"app/node.go:49","msg":"running compaction"}
node-1 | {"level":"info","ts":1711344831.5507004,"caller":"app/node.go:60","msg":"compaction complete"}
node-1 | {"level":"info","ts":1711344831.550728,"caller":"master/master_clock_consensus_engine.go:142","msg":"starting master consensus engine"}
node-1 | {"level":"info","ts":1711344831.550737,"caller":"master/master_clock_consensus_engine.go:149","msg":"syncing last seen state"}
node-1 | {"level":"info","ts":1711344831.616826,"caller":"master/master_clock_consensus_engine.go:179","msg":"subscribing to pubsub messages"}
node-1 | {"level":"info","ts":1711344831.6168523,"caller":"p2p/blossomsub.go:245","msg":"joining broadcast"}
node-1 | {"level":"info","ts":1711344831.6168633,"caller":"p2p/blossomsub.go:254","msg":"subscribe to bitmask","bitmask":"//////////////////////////////////////////8="}
node-1 | {"level":"info","ts":1711344831.6168873,"caller":"p2p/blossomsub.go:261","msg":"begin streaming from bitmask","bitmask":"//////////////////////////////////////////8="}
node-1 | {"level":"info","ts":1711344831.616897,"caller":"master/execution_registration.go:14","msg":"registering execution engine","execution_engine_name":"ceremony"}
node-1 | {"level":"info","ts":1711344831.6169214,"caller":"master/execution_registration.go:23","msg":"starting execution engine at frame","execution_engine_name":"ceremony","current_frame":101682}
node-1 | {"level":"info","ts":1711344831.6169817,"caller":"ceremony/ceremony_execution_engine.go:434","msg":"ceremony data loaded","g2_power":"AwI3CiNfZFrFVJiMleegL9vxKKCuIlC39r0TThV4UvT6tlGd/eDHlpYrS8NS8ecK5YXc1qQjTz31E9PsZE5LRTKdKKdWC6SitKoCyduvXYhPEXqmZujPjCiZY0yqHDFMzvFHRUDORHbH9r72c7NIYnctV8ZQpO1YX771TlB1F0v4tOSqZ85MXxX1STIPTrv34A44Dr2I1TAQK67KoESqW81ccOEm4Cb78hnIKzCRgLlTH38w5GG8hFMoT38xTvJJ2xcYNSa6yxYmEEm9OorQMBjYL/QZAMo1dV0BoQ7oMyivFDrkZIVTXVtjcLbuRIJxHl0F03zlNwVfWjHERPUrCoHflbMD/R3r/0Y1dzmbcKsql73YlADIcs4oO8jAsqW2To0PqJgJanVE2RKM++5zl91X5PQ244kRwkY/WZG+NBU41pQx1Dn7Rup/kRc16Ufllww5QFOQs4DOu0D6vKKZSr9wX8avG1pzo2z+JG+YB8M8/uRXa3MEM1QeUzBFGdI9jWbq0PJqoAzbnYFsu+bwhNbsZf7jeyURKgs6Z5SByDjC0tGNjV/MapPUSQ/QuAKtguaO9ektbASv2XJP95e7iqaje3sE7Q5pKv9NJPuOmT5VR2ONM89kRmlapr8sD4hVS3BS6qbgYzQ7WFtFNkJtzgS4Y00Pya0OSyNRxZbwTawMqUuu4LHQajpwPkOD9j7ndJPF4Jbl2Is7oJSi95DMSYG07mXFng1xwDXDicspef0HjRRzKYJtCeSEq/WmjC4hIP+K0SzMOpju"}
node-1 | {"level":"info","ts":1711344831.6169991,"caller":"ceremony/ceremony_data_clock_consensus_engine.go:265","msg":"starting ceremony consensus engine"}
node-1 | {"level":"info","ts":1711344831.6170056,"caller":"ceremony/ceremony_data_clock_consensus_engine.go:270","msg":"loading last seen state"}
node-1 | {"level":"info","ts":1711344831.61706,"caller":"master/master_clock_consensus_engine.go:202","msg":"peers in store","peer_store_count":164,"network_peer_count":65}
node-1 | panic: get latest data clock frame: get data clock frame: pebble: not found
node-1 |
node-1 | goroutine 363956 [running]:
node-1 | source.quilibrium.com/quilibrium/monorepo/node/consensus/time.(*DataTimeReel).Start(0xc0053d0d80)
node-1 | /opt/ceremonyclient/node/consensus/time/data_time_reel.go:126 +0x2fd
node-1 | source.quilibrium.com/quilibrium/monorepo/node/consensus/ceremony.(*CeremonyDataClockConsensusEngine).Start(0xc0003dbd40)
node-1 | /opt/ceremonyclient/node/consensus/ceremony/ceremony_data_clock_consensus_engine.go:271 +0x96
node-1 | source.quilibrium.com/quilibrium/monorepo/node/execution/intrinsics/ceremony.(*CeremonyExecutionEngine).Start.func1()
node-1 | /opt/ceremonyclient/node/execution/intrinsics/ceremony/ceremony_execution_engine.go:440 +0x33
node-1 | created by source.quilibrium.com/quilibrium/monorepo/node/execution/intrinsics/ceremony.(*CeremonyExecutionEngine).Start
node-1 | /opt/ceremonyclient/node/execution/intrinsics/ceremony/ceremony_execution_engine.go:439 +0x198
node-1 | %#########
node-1 | #############################
node-1 | ########################################&
node-1 | ###############################################
node-1 | &#####################% %######################
node-1 | ################# #################
node-1 | ############### ###############
node-1 | ############# ##############
node-1 | ############# ############&
node-1 | ############ ############
node-1 | ########### ########## &###########
node-1 | ########### ############## ###########
node-1 | ########### ############## ##########&
node-1 | ########## ############## ##########
node-1 | %########## ########## ##########
node-1 | ########## ##########
node-1 | ########## &#########
node-1 | ########## ####### ####### ##########
node-1 | %######### &######################### ##########
node-1 | ########## ##############% ############## &##########
node-1 | ' ' &############## ############### ##########
node-1 | ' ' ############### ##############% ###########
node-1 | ' '. ########## ############### ########
node-1 | '. . ##### ##############% ####
node-1 | ' '. ###############
node-1 | '. '.. ##############%
node-1 | '. '-. ###############
node-1 | '-. ''-.. .. ##############%
node-1 | '-. ''---............----' '. ###############
node-1 | '-.. '. ############
node-1 | ''-.. ..' ########
node-1 | ''---.. ...---'' ##
node-1 | ''----------''
node-1 |
node-1 | Quilibrium Node - v1.4.12 – Sunset
node-1 |
node-1 | Loading ceremony state and starting node...
node-1 | 2024/03/25 05:35:01 [JOB 1] WAL file .config/store/000227.log with log number 000227 stopped reading at offset: 0; replayed 0 keys in 0 batches
node-1 | 2024/03/25 05:35:01 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
node-1 | {"level":"info","ts":1711344901.4872391,"caller":"p2p/blossomsub.go:139","msg":"established peer id","peer_id":"QmdQqnTwyq9hvDJqfTbpR68PApqeKcQML8mKWLWb62aYHp"}
node-1 | {"level":"info","ts":1711344901.4872668,"caller":"p2p/blossomsub.go:332","msg":"establishing dht"}
Hi @zhang-wenchao thanks for reporting this and apologies for the late response! Are you're still experiencing this issue?
Quilibrium Node - v1.4.12 – Sunset Are you really running that version or you are upgrading from it?
