reth icon indicating copy to clipboard operation
reth copied to clipboard

Published 20 hours ago • verified publisher icon paradigmxyz

op-reth - Optimism Mainnet gets stuck at execution stage at 0 blocks and never executes blocks

Open koraykoska opened this issue 1 year ago • 4 comments

Describe the bug

I tried op-reth multiple times and never got it to properly work with Optimism Mainnet. The same setup works with Base Mainnet.

I have a machine with a 14TB lvm ext4 drive (datacenter NVME). We manage nodes through kubernetes (Ubuntu 24.04 on the host machine) and use the official docker containers. The host machine only runs reth and op-node. Some other stats:

  • 3Gbit/s symmetric up/down-link.
  • CPU: AMD EPYC GENOA 9354
  • RAM: 128 GB ECC 4800 MHz

We tested everything from 1.0.0 to 1.0.3. The configuration below is used for reth (for simplicity it's just the bash command and not the kubernetes yaml):

op-reth \
node \
--rollup.sequencer-http=https://mainnet-sequencer.optimism.io \
--rollup.disable-tx-pool-gossip \
--chain=optimism \
--datadir=/reth_db \
--http \
--http.addr=0.0.0.0 \
--http.port=8545 \
--http.corsdomain=* \
--http.api=eth,net,web3,trace,debug,reth,ots,eth-call-bundle \
--ws \
--ws.addr=0.0.0.0 \
--ws.port=8546 \
--ws.origins=* \
--ws.api=eth,net,web3,trace,debug,reth,ots,eth-call-bundle \
--rpc.max-connections=100000 \
--rpc.max-tracing-requests=10000 \
--rpc.max-blocks-per-filter=0 \
--rpc.max-logs-per-response=20000 \
--rpc.gascap=150000000 \
--rpc.max-request-size=30 \
--rpc.max-response-size=200 \
--authrpc.addr=0.0.0.0 \
--authrpc.port=8551 \
--authrpc.jwtsecret=/cl-configs/jwt.hex \
--enable-discv5-discovery \
--nat=publicip \
--addr=0.0.0.0 \
--port=40404 \
--discovery.addr=0.0.0.0 \
--discovery.port=40404 \
--max-outbound-peers=200 \
--max-inbound-peers=200 \
--metrics=0.0.0.0:6060

The config below is op-node (L1_RPC_URL and L1_CL_URL are a reth <> lighthouse combination in the same local network):

op-node \
--l1=$L1_RPC_URL \
--l1.beacon=$L1_CL_URL \
--l1.trustrpc \
--l2=http://127.0.0.1:8551 \
--l2.jwt-secret=/cl-configs/jwt.hex \
--network=op-mainnet \
--rpc.addr=0.0.0.0 \
--rpc.port=9545 \
--rollup.load-protocol-versions=true \
--rollup.halt=major \
--syncmode=execution-layer

Again, the same configuration works with base mainnet but not with Optimism Mainnet. There are no apparent errors in the logs of op-node and op-reth. Everything up to execution seems to work (even though it also requires some manual restarts here and there).

The example below is a newly newly restarted op-reth, but it can go on for hours without ever executing a single block. Even restarting doesn't fix the problem.

2024-07-25T10:24:03.867384Z  INFO Status connected_peers=9 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:24:28.867497Z  INFO Status connected_peers=9 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:24:53.867154Z  INFO Status connected_peers=10 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:25:18.867809Z  INFO Status connected_peers=10 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:25:43.867601Z  INFO Status connected_peers=10 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:26:08.867516Z  INFO Status connected_peers=10 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:26:33.868091Z  INFO Status connected_peers=12 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:26:58.867797Z  INFO Status connected_peers=12 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:27:23.867521Z  INFO Status connected_peers=12 freelist=7 stage=Execution checkpoint=0 target=1230665122024-07-25T10:27:48.867666Z  INFO Status connected_peers=12 freelist=7 stage=Execution checkpoint=0 target=123066512

Steps to reproduce

  1. Setup an Ubuntu 24.04 server with similar configs to the one mentioned above.
  2. Run op-node and op-reth with the configs as above.
  3. Wait until execution.
  4. See that it hangs.

Node logs

2024-07-25T10:42:17.735567Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:19.662316Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:21.467462Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:23.161673Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:23.867282Z  INFO reth::cli: Status connected_peers=20 freelist=7 stage=Execution checkpoint=0 target=123066512
2024-07-25T10:42:25.494036Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:27.284657Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:29.265239Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:30.735024Z DEBUG net: Session established remote_addr=3.10.19.215:30308 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0xfea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7 total_active=21 kind=outgoing peer_enode=enode://fea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7@3.10.19.215:30308
2024-07-25T10:42:30.755808Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xfea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7
2024-07-25T10:42:30.912426Z DEBUG net: Session established remote_addr=3.10.19.215:30311 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x22c8ea45116d19e4ebc4bbb59be76d84f4fe0215c7d924fbae68d7c85ea8d685feb8805095ce94130d9cd4fe153bcb2a8494a3c6b5b06d785d74e6934439a0cc total_active=21 kind=outgoing peer_enode=enode://22c8ea45116d19e4ebc4bbb59be76d84f4fe0215c7d924fbae68d7c85ea8d685feb8805095ce94130d9cd4fe153bcb2a8494a3c6b5b06d785d74e6934439a0cc@3.10.19.215:30311
2024-07-25T10:42:30.933225Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x22c8ea45116d19e4ebc4bbb59be76d84f4fe0215c7d924fbae68d7c85ea8d685feb8805095ce94130d9cd4fe153bcb2a8494a3c6b5b06d785d74e6934439a0cc
2024-07-25T10:42:31.064691Z DEBUG net::discv5: connected peers in routing table connected_peers=162
2024-07-25T10:42:31.662390Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:32.650004Z  INFO reth::cli: Initialized tracing, debug log directory: /root/.cache/reth/logs/optimism
2024-07-25T10:42:33.160808Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:35.471279Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:35.671833Z DEBUG net::session: pending session timed out remote_addr=15.204.198.129:30303 direction=Outgoing(0xec9dca0037147f539651209bacf4a1d1e779d7a2f970a1eefd4df699877ca1fa1204408f0ba444b7661a40f1285758fa306137b2cf91f573b3c105b06885050b)
2024-07-25T10:42:35.956775Z DEBUG net::session: pending session timed out remote_addr=153.160.98.71:51849 direction=Outgoing(0x6ce1f059df1ad74f642eaab5dfa822c93616912806f58340532195e99f144da345f0d4d2f668fee38abfee8bf3f951a61fb9cc40e89be1a5aefcc4e106fed51a)
2024-07-25T10:42:37.161971Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:39.471762Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:41.915960Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:43.163908Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:45.722934Z DEBUG net: Session established remote_addr=3.10.19.215:30310 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x6b5af7bc909b209c2a3be9bbb5a11bb49aa833dd62e8f153c38c0ccea5e5090c5604a99f7c92b072792357f5ad803ea62ee69115378417281a3515e7b5543936 total_active=21 kind=outgoing peer_enode=enode://6b5af7bc909b209c2a3be9bbb5a11bb49aa833dd62e8f153c38c0ccea5e5090c5604a99f7c92b072792357f5ad803ea62ee69115378417281a3515e7b5543936@3.10.19.215:30310
2024-07-25T10:42:45.739686Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x6b5af7bc909b209c2a3be9bbb5a11bb49aa833dd62e8f153c38c0ccea5e5090c5604a99f7c92b072792357f5ad803ea62ee69115378417281a3515e7b5543936
2024-07-25T10:42:45.747826Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:45.784522Z DEBUG net: Session established remote_addr=3.10.19.215:30307 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827 total_active=21 kind=outgoing peer_enode=enode://25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827@3.10.19.215:30307
2024-07-25T10:42:45.804389Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827
2024-07-25T10:42:47.674527Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:48.867846Z  INFO reth::cli: Status connected_peers=20 freelist=7 stage=Execution checkpoint=0 target=123066512
2024-07-25T10:42:49.164005Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:51.851010Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:53.547930Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:55.547837Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:57.546869Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:42:59.551002Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:00.763653Z DEBUG net: Session established remote_addr=3.10.19.215:30303 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x6c35ff4edc1be29d8f9bb7c16ffc8773eafdd488b2239dd5b8400c4a5f723cef473b7423a397cb76dfa368e943c15a7b5f943936faff79e4ce28c64a44ec3eb7 total_active=21 kind=outgoing peer_enode=enode://6c35ff4edc1be29d8f9bb7c16ffc8773eafdd488b2239dd5b8400c4a5f723cef473b7423a397cb76dfa368e943c15a7b5f943936faff79e4ce28c64a44ec3eb7@3.10.19.215:30303
2024-07-25T10:43:00.777745Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x6c35ff4edc1be29d8f9bb7c16ffc8773eafdd488b2239dd5b8400c4a5f723cef473b7423a397cb76dfa368e943c15a7b5f943936faff79e4ce28c64a44ec3eb7
2024-07-25T10:43:01.261333Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:03.737021Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:05.263776Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:07.547258Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:09.582718Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:12.237149Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:13.554434Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:13.867508Z  INFO reth::cli: Status connected_peers=20 freelist=7 stage=Execution checkpoint=0 target=123066512
2024-07-25T10:43:15.484023Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:15.719439Z DEBUG net: Session established remote_addr=3.10.19.215:30308 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0xfea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7 total_active=21 kind=outgoing peer_enode=enode://fea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7@3.10.19.215:30308
2024-07-25T10:43:15.735226Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xfea359e2843059d4f99bb09da45400a6b3f145e80218a1769b86c31362269795cccb1981e87733e45ce4300af01ec524ba28344df293823a262f6add3cfa7af7
2024-07-25T10:43:15.891471Z DEBUG net: Session established remote_addr=3.10.19.215:30311 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x22c8ea45116d19e4ebc4bbb59be76d84f4fe0215c7d924fbae68d7c85ea8d685feb8805095ce94130d9cd4fe153bcb2a8494a3c6b5b06d785d74e6934439a0cc total_active=21 kind=outgoing peer_enode=enode://22c8ea45116d19e4ebc4bbb59be76d84f4fe0215c7d924fbae68d7c85ea8d685feb8805095ce94130d9cd4fe153bcb2a8494a3c6b5b06d785d74e6934439a0cc@3.10.19.215:30311
2024-07-25T10:43:15.905802Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x22c8ea45116d19e4ebc4bbb59be76d84f4fe0215c7d924fbae68d7c85ea8d685feb8805095ce94130d9cd4fe153bcb2a8494a3c6b5b06d785d74e6934439a0cc
2024-07-25T10:43:18.057328Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:18.347297Z  INFO reth::cli: Initialized tracing, debug log directory: /root/.cache/reth/logs/optimism
2024-07-25T10:43:19.741119Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:21.570651Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:23.415713Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:25.554838Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:27.416067Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:29.413961Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:30.732561Z DEBUG net: Session established remote_addr=3.10.19.215:30310 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x6b5af7bc909b209c2a3be9bbb5a11bb49aa833dd62e8f153c38c0ccea5e5090c5604a99f7c92b072792357f5ad803ea62ee69115378417281a3515e7b5543936 total_active=21 kind=outgoing peer_enode=enode://6b5af7bc909b209c2a3be9bbb5a11bb49aa833dd62e8f153c38c0ccea5e5090c5604a99f7c92b072792357f5ad803ea62ee69115378417281a3515e7b5543936@3.10.19.215:30310
2024-07-25T10:43:30.752551Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x6b5af7bc909b209c2a3be9bbb5a11bb49aa833dd62e8f153c38c0ccea5e5090c5604a99f7c92b072792357f5ad803ea62ee69115378417281a3515e7b5543936
2024-07-25T10:43:30.838496Z DEBUG net: Session established remote_addr=3.10.19.215:30307 client_version=Nodecrawler/v1.14.0-unstable/linux-amd64/go1.22.3 peer_id=0x25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827 total_active=21 kind=outgoing peer_enode=enode://25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827@3.10.19.215:30307
2024-07-25T10:43:30.858329Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x25b663f4199eee2490ff82110fce5fc2051d3808a18bb79e7ee410bdddca18e67bfb5710ddebb8469a36fa6867d374c0c0f89aa40ba2ff311a3e87ee9fe6b827
2024-07-25T10:43:31.751304Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:33.491024Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:35.417965Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:36.261958Z DEBUG net::discv5: connected peers in routing table connected_peers=178
2024-07-25T10:43:37.550302Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-07-25T10:43:38.867641Z  INFO reth::cli: Status connected_peers=20 freelist=7 stage=Execution checkpoint=0 target=123066512

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 1.0.3 Commit SHA: 390f30aadebcdd509e72cc04327c3b854de076a6 Build Timestamp: 2024-07-18T16:45:06.971621164Z Build Features: jemalloc,optimism Build Profile: maxperf

What database version are you on?

Current database version: 2 Local database version: 2

Which chain / network are you on?

Optimism Mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

  • [X] I agree to follow the Code of Conduct

koraykoska avatar Jul 25 '24 10:07 koraykoska