developers icon indicating copy to clipboard operation
developers copied to clipboard
verified publisher icon ethereum-optimism

Problems after migrating to the Fusaka network. Errors in the op-node

Open neodiz opened this issue 1 month ago • 7 comments

Bug Description I have a network connected to Sepolia. Transactions are stuck in pending mode. Restarting op-batcher temporarily solves the problem. I also see an error in the op-node logs. Op-batcher also has a warning message

Steps to Reproduce The problem started after updating Kusaka Expected behavior

Environment Information:

  • Ubuntu 22.04
  • docker version 27.5.1
  • op-geth Geth/v1.101603.1-stable-8da5bf08/linux-amd64/go1.24.
  • op-node version v0.0.0-c1081e3a-1759886077(14.1) build from source
  • op-batcher version v1.16.0-64cb0d24-1758896861

Configurations:

    image: "repo.mind-dev.com/sanchain/op-node:v1.14.1"
    command: 
      - op-node
      - --l2=http://geth:8551 
      - --l2.jwt-secret=/jwtsecret
      - --sequencer.enabled 
      - --l1.trustrpc=true
      - --sequencer.l1-confs=5 
      - --verifier.l1-confs=4 
      - --rollup.config=/files/rollup.json 
      - --rpc.addr=0.0.0.0
      - --rpc.enable-admin 
      - --p2p.sequencer.key=$SEQUENSER_KEY
      - --l1=$L1_NODE
      - --l1.beacon=$L1_BEACON
      - --log.level=Debug
      - --p2p.listen.tcp=9222
      - --p2p.listen.udp=9222
      - --p2p.advertise.ip=$EXTERNAL_IP
      - --p2p.bootnodes=$BOOTNODE_NODE
      - --safedb.path=/db
      - --metrics.enabled
      - --metrics.port=9100
      - --metrics.addr=0.0.0.0
      
  op-batcher
    image: "us-docker.pkg.dev/oplabs-tools-artifacts/images/op-batcher:64cb0d2490fea1b00dcce80aa2f39bafc47a2010"
    command:
      - op-batcher
      - --l2-eth-rpc=http://geth:8545 
      - --rollup-rpc=http://op-node:9545 
      - --poll-interval=1s 
      - --sub-safety-margin=6 
      - --num-confirmations=1 
      - --safe-abort-nonce-too-low-count=3 
      - --resubmission-timeout=30s 
      - --rpc.addr=0.0.0.0 
      - --rpc.port=8548 
      - --rpc.enable-admin 
      - --max-channel-duration=150
      - --target-num-frames=5
      - --txmgr.min-basefee=2
      - --txmgr.min-tip-cap=2
      - --batch-type=1
      - --compression-algo=brotli-11 
      - --data-availability-type=blobs 
      - --approx-compr-ratio=0.9 
      - --max-l1-tx-size-bytes=240000 
      - --l1-eth-rpc=$L1_NODE 
      - --private-key=$BATCHER_KEY
      - --log.level=debug
      - --metrics.enabled
      - --metrics.port=9100
      - --metrics.addr=0.0.0.0

Logs:

op-node 
t=2025-10-23T07:55:09+0000 lvl=warn msg="Engine temporary error" err="derivation failed: temp: failed to fetch blobs: expected 2 blobs but got 8"
t=2025-10-23T07:55:09+0000 lvl=debug msg="scheduling re-attempt with delay" attempts=4997 delay=10.21348529s
t=2025-10-23T07:55:09+0000 lvl=error msg="Engine failed temporarily, backing off sequencer" err="derivation failed: temp: failed to fetch blobs: expected 2 blobs but got 8"
t=2025-10-23T07:55:09+0000 lvl=debug msg="Sequencer action schedule changed" t=2025-10-23T07:55:10+0000 wait=999.99987ms ok=true event=engine-temporary-error

Additional context The problem started after updating Kusaka

neodiz avatar Oct 23 '25 08:10 neodiz

Image

neodiz avatar Oct 23 '25 08:10 neodiz

op-batcher logs

t=2025-10-23T08:22:01+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406210 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:05+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406350 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:07+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406420 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:09+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406490 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:11+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406560 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:13+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406630 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:18+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406870 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:19+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15406940 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:21+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407010 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:23+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407080 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:25+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407250 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:30+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407390 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:31+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407460 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:33+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407530 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:35+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407600 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:37+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407770 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:42+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15407840 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:43+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408080 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:45+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408150 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:47+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408220 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:49+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408390 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:54+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408460 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:55+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408700 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:57+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408770 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:22:59+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15408840 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:01+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409010 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:06+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409250 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:07+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409320 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:09+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409390 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:11+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409560 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:13+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409630 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:18+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409870 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:19+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15409940 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:21+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15410010 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:23+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15410180 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic
t=2025-10-23T08:23:25+0000 lvl=warn msg="Throttling loop: unsafe bytes above threshold, scaling endpoint throttling based on intensity" unsafe_bytes=15410250 intensity=1 max_tx_size=150 max_block_size=2000 controller_type=quadratic

neodiz avatar Oct 23 '25 08:10 neodiz

t=2025-10-23T07:55:09+0000 lvl=warn msg="Engine temporary error" err="derivation failed: temp: failed to fetch blobs: expected 2 blobs but got 8", I'm not sure if this is a issue of the beacon node, could you please provide more information about the L1 beacon node, which client and the version

jsvisa avatar Oct 27 '25 08:10 jsvisa

I have 2 node.

  1. connected now
    • prysm (prysm-beacon-chain:v6.1.2)
    • geth ( Geth/v1.16.6-unstable-116c9167-20251022/linux-amd64/go1.24.9)
  2. backup node
  • lighhouse (Lighthouse v8.0.0-rc.0-e5b4983)
  • erigon (erigon version 3.2.0-3418e071)

neodiz avatar Oct 27 '25 08:10 neodiz

Did you follow all the steps laid out here: https://docs.optimism.io/notices/fusaka-notice

sbvegan avatar Oct 29 '25 16:10 sbvegan

op-geth configuration

geth:
    image: "repo.mind-dev.com/sanchain/op-geth:v1.14.0"
    stop_grace_period: 300s
    command:
      - --http
      - --http.api=web3,debug,eth,txpool,net,engine,miner
      - --http.addr=0.0.0.0
      - --authrpc.vhosts=*
      - --http.vhosts=*
      - --http.corsdomain=*
      - --authrpc.addr=0.0.0.0
      - --authrpc.vhosts=*
      - --authrpc.port=8551
      - --ws
      - --ws.addr=0.0.0.0 
      - --ws.port=8546 
      - --ws.origins=* 
      - --ws.api=debug,eth,txpool,net,engine,miner
      - --authrpc.jwtsecret=/jwtsecret
      - --datadir=/data
      - --state.scheme=hash
      - --syncmode=full
      - --gcmode=archive
      - --networkid=${CHAIN_ID:-42093}
      - --nat=extip:$EXTERNAL_IP
      - --port=$GETH_P2P_PORT
      - --rollup.disabletxpoolgossip=true
      - --bootnodes=$BOOTNODE_GETH
      - --log.file=/logs/geth.log
      - --gpo.ignoreprice=1
      - --miner.gasprice=1
      - --txpool.pricelimit=1
      - --metrics
      - --metrics.port=9100
      - --metrics.addr=0.0.0.0

neodiz avatar Oct 29 '25 17:10 neodiz

Have you any news ?

neodiz avatar Nov 18 '25 07:11 neodiz