stacks-core
stacks-core copied to clipboard
stacks-network
node halts [mainnet]
Stacks node: 2.05.0.6.0
When syncing from genesis, the stacks node abruptly comes to a halt. Upon looking at the logs, it looks like there are some Atlas or BNS related errors. While I think simply restarting the node will get past this issue, I want to make sure this issue is reported so someone can take a look.
12/21/2022, 10:36:21 AM | Atlas: 1 attachment instances emitted from events
-- | --
| | | 12/21/2022, 10:36:22 AM | Atlas: could not find a peer including attachment in its inventory
| | | 12/21/2022, 10:36:22 AM | Atlas: re-enqueuing batch AttachmentsBatch { stacks_block_height: 88066, canonical_stacks_tip_height: Some(88066), index_block_hash: c6b9aff11a7a2c7a2d3b30a7878eb3a27e1291e6e203bc890f8c44fee19bdeb5, attachments_instances: {QualifiedContractIdentifier { issuer: StandardPrincipalData(SP000000000000000000002Q6VF78), name: ContractName("bns") }: {223943: be4b34eedd34bc6034fc6f2aa4466174e58a378d}}, retry_count: 1, retry_deadline: 1671647784 } for retry
2x | | | 12/21/2022, 10:36:26 AM | Handle HTTPRequest
| | | 12/21/2022, 10:36:31 AM | Atlas: could not find a peer including attachment in its inventory
| | | 12/21/2022, 10:36:31 AM | Atlas: re-enqueuing batch AttachmentsBatch { stacks_block_height: 88066, canonical_stacks_tip_height: Some(88066), index_block_hash: c6b9aff11a7a2c7a2d3b30a7878eb3a27e1291e6e203bc890f8c44fee19bdeb5, attachments_instances: {QualifiedContractIdentifier { issuer: StandardPrincipalData(SP000000000000000000002Q6VF78), name: ContractName("bns") }: {223943: be4b34eedd34bc6034fc6f2aa4466174e58a378d}}, retry_count: 2, retry_deadline: 1671647795 } for retry
Is the node still running? Can you attach gdb and give me stack trace for each thread?
@deantchi did restart help in your case? I'm having similar issue with same signature and I've restarted the node multiple times but it only advances 1-2 blocks before halting again.
stacks-blockchain | WARN [1672979450.044097] [src/net/atlas/download.rs:436] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Atlas: could not find a peer including attachment in its inventory
stacks-blockchain | INFO [1672979450.324735] [src/net/atlas/download.rs:218] [p2p-(0.0.0.0:20444,0.0.0.0:20443)] Atlas: re-enqueuing batch AttachmentsBatch { stacks_block_height: 89992, canonical_stacks_tip_height: Some(89992), index_block_hash: 349b0ce86939fbfdb24073868be450287bbd0fe014bcb6bbfae18b39d8d11157, attachments_instances: {QualifiedContractIdentifier { issuer: StandardPrincipalData(SP000000000000000000002Q6VF78), name: ContractName("bns") }: {227050: e454ab15ec9aee2bb3695a8db4e86a1074357bef}}, retry_count: 8, retry_deadline: 1672979751 } for retry
I can provide logs as needed just need instructions. Here are my stacks-blockchain docker logs: https://transfer.sh/fl0Ipb/sb.tar.gz
@pseudozach I'm seeing the same behavior with reboots. I'm still trying to get thread traces from my affected node. We are running our node in a more restricted environment so getting GDB traces on the node is a challenge.
@pseudozach I'm seeing the same behavior with reboots. I'm still trying to get thread traces from my affected node. We are running our node in a more restricted environment so getting GDB traces on the node is a challenge.
in my case the root cause turned out to be foundation's btc node being overloaded. so I switched to my own btc node as burnchain node and issue seems to have resolved for now.
I think this is blocked on #3086. Assigning to @deantchi for now. Please feel free to re-assign or close as you see fit.
We're experiencing the same issue. Still no clue why the node halts from time to time.
