stuck at SenderRecovery

[mrabino1]

Describe the bug

see logs below. seemingly stuck at SenderRecovery

Steps to reproduce

install beta.6 .. sync full from start.. wait

Node logs

$ tail -f reth.log 
2024-04-25T05:30:37.727450Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:37.732667Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:30:37.732880Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:37.869789Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:37.870105Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:38.601058Z  INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=unknown
2024-04-25T05:30:40.224789Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:44.239314Z DEBUG discv4: Failed to send datagram. to=128.10.126.23:0 err=Invalid argument (os error 22)
2024-04-25T05:30:44.602015Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:45.421767Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.227048Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.230649Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:30:49.230851Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.361554Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.361853Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:52.226295Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.056854Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.059692Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:01.059895Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.251420Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.251760Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:03.601077Z  INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=unknown
2024-04-25T05:31:04.228211Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.415023Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.415960Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.785010Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.789528Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:12.789777Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.923689Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.924015Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:16.228813Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.111408Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.114043Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:25.114300Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.244952Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.245304Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:28.229315Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:28.601081Z  INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=unknown
2024-04-25T05:31:37.210277Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:37.213663Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:37.213882Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:37.346096Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:37.346430Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:40.231135Z DEBUG jsonrpsee-server: Accepting new connection 1/500

Platform(s)

Linux (x86)

What version/commit are you on?

beta.6

What database version are you on?

beta.6

Which chain / network are you on?

mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• [X] I agree to follow the Code of Conduct

[mrabino1]

a joyful sudo systemctl restart reth.service

has seemingly kicked it.. and allowed it to continue..

[mrabino1]

here are the logs right before things stopped progressing... fwiw

2024-04-24T20:01:08.442017Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333025538..1333125538 2024-04-24T20:01:08.442030Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333125538..1333225538 2024-04-24T20:01:08.442184Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333125538..1333225538 2024-04-24T20:01:09.583157Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333125538..1333225538 2024-04-24T20:01:09.583170Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333225538..1333325538 2024-04-24T20:01:09.583305Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333225538..1333325538 2024-04-24T20:01:10.680326Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333225538..1333325538 2024-04-24T20:01:10.680339Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333325538..1333425538 2024-04-24T20:01:10.680561Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333325538..1333425538 2024-04-24T20:01:11.747709Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333325538..1333425538 2024-04-24T20:01:11.747720Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333425538..1333525538 2024-04-24T20:01:11.747856Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333425538..1333525538 2024-04-24T20:01:12.421821Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:12.422400Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:12.709291Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:12.781587Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:12.911447Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile" 2024-04-24T20:01:12.911740Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:12.914792Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333425538..1333525538 2024-04-24T20:01:12.914800Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333525538..1333625538 2024-04-24T20:01:12.914954Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333525538..1333625538 2024-04-24T20:01:13.189575Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:13.193612Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:14.284530Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333525538..1333625538 2024-04-24T20:01:14.284543Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333625538..1333725538 2024-04-24T20:01:14.284737Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333625538..1333725538 2024-04-24T20:01:14.841526Z DEBUG net::session: pending session timed out remote_addr=213.225.253.90:30303 direction=Outgoing(0x40456c1d52acecaa4f964110be1a207d99f1af8b98022d6e5b48f41c1010e7f0226ea2f406d2d0008dc68abe119436a298b35a08d17516ff578170db2159c168) 2024-04-24T20:01:15.682204Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333625538..1333725538 2024-04-24T20:01:15.682216Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333725538..1333825538 2024-04-24T20:01:15.682358Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333725538..1333825538 2024-04-24T20:01:16.913738Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333725538..1333825538 2024-04-24T20:01:16.913751Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333825538..1333925538 2024-04-24T20:01:16.913917Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333825538..1333925538 2024-04-24T20:01:18.202049Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333825538..1333925538 2024-04-24T20:01:18.202062Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333925538..1334025538 2024-04-24T20:01:18.202241Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333925538..1334025538 2024-04-24T20:01:19.601992Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333925538..1334025538 2024-04-24T20:01:19.602005Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334025538..1334125538 2024-04-24T20:01:19.602199Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334025538..1334125538 2024-04-24T20:01:20.577321Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x15c9b7dd86939afdcff892f85e70b99d3733877ecc44ed1e1b865f65a02bb44693ec2cae2148f51803823542e7728a9996c365cfbc37bb72ef55bee7e4fc9a95 2024-04-24T20:01:20.894949Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334025538..1334125538 2024-04-24T20:01:20.894962Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334125538..1334225538 2024-04-24T20:01:20.895141Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334125538..1334225538 2024-04-24T20:01:22.163950Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334125538..1334225538 2024-04-24T20:01:22.163963Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334225538..1334325538 2024-04-24T20:01:22.164252Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334225538..1334325538 2024-04-24T20:01:23.363315Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334225538..1334325538 2024-04-24T20:01:23.363329Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334325538..1334425538 2024-04-24T20:01:23.363500Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334325538..1334425538 2024-04-24T20:01:24.577785Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334325538..1334425538 2024-04-24T20:01:24.577797Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334425538..1334525538 2024-04-24T20:01:24.578004Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334425538..1334525538 2024-04-24T20:01:24.724788Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:25.312298Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:25.904397Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile" 2024-04-24T20:01:25.905547Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:25.906369Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334425538..1334525538 2024-04-24T20:01:25.906377Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334525538..1334625538 2024-04-24T20:01:25.906611Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334525538..1334625538 2024-04-24T20:01:26.289952Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:26.293731Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:27.286999Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334525538..1334625538 2024-04-24T20:01:27.287013Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334625538..1334725538 2024-04-24T20:01:27.287340Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334625538..1334725538 2024-04-24T20:01:28.602029Z INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=3h 24m 24s 2024-04-24T20:01:28.614645Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334625538..1334725538 2024-04-24T20:01:28.614654Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334725538..1334825538 2024-04-24T20:01:28.614947Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334725538..1334825538 2024-04-24T20:01:36.503277Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:36.508943Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile" 2024-04-24T20:01:36.509133Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:36.637294Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:36.637611Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:36.712185Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:48.711811Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:48.713391Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:48.715257Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile" 2024-04-24T20:01:48.715423Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:48.839259Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:48.839587Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:01:53.601559Z INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=3h 23m 59s 2024-04-24T20:02:00.715043Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:02:01.251175Z DEBUG jsonrpsee-server: Accepting new connection 1/500 2024-04-24T20:02:01.254890Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"

[emhane]

you can try to postpone pruning https://github.com/paradigmxyz/reth/issues/7500#issuecomment-2074698780. I would also turn of the rpc server for initial sync. @mrabino1

[github-actions[bot]]

This issue is stale because it has been open for 21 days with no activity.

[emhane]

did syncing work out for you in the end @mrabino1 ?

[mrabino1]

y. i will close. looking forward this being out of beta !