Unable to sync after upgrade

[B1Q]

just upgraded to the latest update and it's throwing a bunch of warnings and never catching up I am running into this warning and it's just counting down L1 block if i leave it to get to 0 it will take a couple years t=2024-05-06T23:52:22+0000 lvl=info msg="Walking back L1Block by hash" curr=0x642778123c963e826fc06e90dfad13fed1cc87de633be4a4b7215d6d73822707:19709592 next=

[rabbitprincess]

i have same issue

[B1Q]

i have same issue

join the queue

[RyanWang0811]

same issue

[rabbitprincess]

it can sync after downgrade op-node to v1.7.1. there are a lot of changes in optimism like blob, snap sync, fault proof between v1.7.1 and v1.7.5.

[B1Q]

t=2024-05-07T22:29:58+0000 lvl=warn msg="Derivation process temporary error" attempts=3911 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x56df86f88d654cfca4a7e72151e61789b84906e8b77a9f720b27ccf3fcc3f000:19521297: failed to fetch blob sidecars for slot 8722070 block 0x56df86f88d654cfca4a7e72151e61789b84906e8b77a9f720b27ccf3fcc3f000:19521297: failed request with status 500: {"code":500,"message":"block not found"}"

Now I am getting these errors, good god.

[B1Q]

it can sync after downgrade op-node to v1.7.1. there are a lot of changes in optimism like blob, snap sync, fault proof between v1.7.1 and v1.7.5.

tried this and it didn't work unfortunately.

[CryptoSailors]

I updated a node and now have this

eth-1  | INFO [05-08|08:52:17.380] Loaded most recent local block           number=14,184,271 hash=b88684..b42135 td=0 age=7m28s
geth-1  | INFO [05-08|08:52:17.380] Loaded most recent local finalized block number=14,183,720 hash=55d9d8..9e48ab td=0 age=25m50s
geth-1  | WARN [05-08|08:52:17.382] Head state missing, repairing            number=14,184,271 hash=b88684..b42135 snaproot=614833..66c538

How long need to wait?

[B1Q]

I updated a node and now have this

eth-1  | INFO [05-08|08:52:17.380] Loaded most recent local block           number=14,184,271 hash=b88684..b42135 td=0 age=7m28s
geth-1  | INFO [05-08|08:52:17.380] Loaded most recent local finalized block number=14,183,720 hash=55d9d8..9e48ab td=0 age=25m50s
geth-1  | WARN [05-08|08:52:17.382] Head state missing, repairing            number=14,184,271 hash=b88684..b42135 snaproot=614833..66c538

How long need to wait?

Give it a week at most If it doesn’t get solved just scrap your data I guess

[parsdextra]

node-geth-1 | INFO [05-09|01:37:05.900] Aborting state snapshot generation root=1a9ed5..5e9a6c in=be67e4..e9fca9 at=6ef3ef..a08802 accounts=50,281,878 slots=249,458,569 storage=20.61GiB dangling=0 elapsed=45m40.536s eta=15m44.105s node-geth-1 | INFO [05-09|01:37:05.903] Resuming state snapshot generation root=e1aedd..549099 in=be67e4..e9fca9 at=6ef3ef..a08802 accounts=50,281,878 slots=249,458,569 storage=20.61GiB dangling=0 elapsed=45m40.539s eta=15m44.106s node-geth-1 | INFO [05-09|01:37:05.912] Imported new potential chain segment number=14,214,639 hash=4f03e3..d4a901 blocks=1 txs=48 mgas=5.597 elapsed=65.346ms mgasps=85.646 snapdiffs=167.92KiB triedirty=310.61MiB node-geth-1 | INFO [05-09|01:37:05.915] Chain head was updated number=14,214,639 hash=4f03e3..d4a901 root=a0b0d7..fea78f elapsed=2.936935ms node-node-1 | t=2024-05-09T01:37:05+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer" l2_finalized=0x2fbafacdebe4ac05e6e9e496a4ef73994930a44a63c6ad33fadfd9858d731101:14214097 l2_safe=0x3a495006a3f2124aee710c4a395ba64af907933b19a1a31955a031b1524c9b28:14214549 l2_pending_safe=0x3a495006a3f2124aee710c4a395ba64af907933b19a1a31955a031b1524c9b28:14214549 l2_unsafe=0x4f03e3e132983523f855da6a025e5146d0e2df315b7e6aba1bb552e48dd4a901:14214639 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1715218625 l1_derived=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 node-node-1 | t=2024-05-09T01:37:05+0000 lvl=debug msg="Derivation process step" onto_origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 attempts=0 node-node-1 | t=2024-05-09T01:37:05+0000 lvl=debug msg="can't find next L1 block info (yet)" number=19829249 origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 node-node-1 | t=2024-05-09T01:37:05+0000 lvl=debug msg="Derivation process went idle" progress=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 err=EOF node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="validated blockv3" topic=blocksV3 result=ACCEPT from=16Uiu2HAmSYvBi6hczyhZaM41q2bjqxPUnquCxm1CBX4MKahMezC4 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=info msg="Received signed execution payload from p2p" id=0x5b9fdd3b6cef7bb97e4018710cd6e8fccdf81eb194f2e5ccb3c54403e42c5842:14214640 peer=16Uiu2HAmSYvBi6hczyhZaM41q2bjqxPUnquCxm1CBX4MKahMezC4 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x5b9fdd3b6cef7bb97e4018710cd6e8fccdf81eb194f2e5ccb3c54403e42c5842:14214640 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="Derivation process step" onto_origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 attempts=0 node-geth-1 | INFO [05-09|01:37:07.710] Aborting state snapshot generation root=e1aedd..549099 in=be67e4..e9fca9 at=6f49f8..c0adf9 accounts=50,281,878 slots=249,795,466 storage=20.63GiB dangling=0 elapsed=45m42.345s eta=15m44.728s node-geth-1 | INFO [05-09|01:37:07.713] Resuming state snapshot generation root=4aab8d..7e2a17 in=be67e4..e9fca9 at=6f49f8..c0adf9 accounts=50,281,878 slots=249,795,466 storage=20.63GiB dangling=0 elapsed=45m42.348s eta=15m44.729s node-geth-1 | INFO [05-09|01:37:07.722] Imported new potential chain segment number=14,214,640 hash=5b9fdd..2c5842 blocks=1 txs=47 mgas=7.576 elapsed=123.701ms mgasps=61.245 snapdiffs=159.73KiB triedirty=310.61MiB node-geth-1 | INFO [05-09|01:37:07.727] Chain head was updated number=14,214,640 hash=5b9fdd..2c5842 root=5d0d81..8190c6 elapsed=3.52953ms node-node-1 | t=2024-05-09T01:37:07+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer" l2_finalized=0x2fbafacdebe4ac05e6e9e496a4ef73994930a44a63c6ad33fadfd9858d731101:14214097 l2_safe=0x3a495006a3f2124aee710c4a395ba64af907933b19a1a31955a031b1524c9b28:14214549 l2_pending_safe=0x3a495006a3f2124aee710c4a395ba64af907933b19a1a31955a031b1524c9b28:14214549 l2_unsafe=0x5b9fdd3b6cef7bb97e4018710cd6e8fccdf81eb194f2e5ccb3c54403e42c5842:14214640 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1715218627 l1_derived=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="Derivation process step" onto_origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 attempts=0 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="can't find next L1 block info (yet)" number=19829249 origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="Derivation process went idle" progress=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 err=EOF node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="peering tick" connected=25 advertised_udp=9222 advertised_tcp=9222 advertised_ip=174.138.185.246 node-node-1 | t=2024-05-09T01:37:07+0000 lvl=debug msg="discovered peer" peer=16Uiu2HAm6SPCMiTW6fAw6JaYtj2hhGowE4ShXpMmHAGVFyF9YKEM nodeID=a914b68bb62198c22a205d515484bcce16946710dc5cdb7ba7979ddc00c33880 addr=/ip4/63.250.53.74/tcp/9222 node-node-1 | t=2024-05-09T01:37:09+0000 lvl=debug msg="validated blockv3" topic=blocksV3 result=ACCEPT from=16Uiu2HAmGNDrgPDErfq4FEgLRjnASJYtpm8FP5AG3PmqLuEPqgxE node-node-1 | t=2024-05-09T01:37:09+0000 lvl=info msg="Received signed execution payload from p2p" id=0xf467918f43f50414718d2cee3c7cb272a53e096a0eea22f551b9552d18cd18ad:14214641 peer=16Uiu2HAmGNDrgPDErfq4FEgLRjnASJYtpm8FP5AG3PmqLuEPqgxE node-node-1 | t=2024-05-09T01:37:09+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xf467918f43f50414718d2cee3c7cb272a53e096a0eea22f551b9552d18cd18ad:14214641 node-node-1 | t=2024-05-09T01:37:09+0000 lvl=debug msg="Derivation process step" onto_origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 attempts=0 node-geth-1 | INFO [05-09|01:37:09.615] Aborting state snapshot generation root=4aab8d..7e2a17 in=be67e4..e9fca9 at=6fa33e..be95dd accounts=50,281,878 slots=250,144,651 storage=20.66GiB dangling=0 elapsed=45m44.251s eta=15m45.385s node-geth-1 | INFO [05-09|01:37:09.618] Resuming state snapshot generation root=f42fcf..68b013 in=be67e4..e9fca9 at=6fa33e..be95dd accounts=50,281,878 slots=250,144,651 storage=20.66GiB dangling=0 elapsed=45m44.254s eta=15m45.386s node-geth-1 | INFO [05-09|01:37:09.627] Imported new potential chain segment number=14,214,641 hash=f46791..cd18ad blocks=1 txs=40 mgas=4.848 elapsed=56.611ms mgasps=85.644 snapdiffs=150.34KiB triedirty=310.47MiB node-geth-1 | INFO [05-09|01:37:09.631] Chain head was updated number=14,214,641 hash=f46791..cd18ad root=15258e..dee7fb elapsed=3.323748ms node-node-1 | t=2024-05-09T01:37:09+0000 lvl=info msg="Sync progress" reason="unsafe payload from sequencer" l2_finalized=0x2fbafacdebe4ac05e6e9e496a4ef73994930a44a63c6ad33fadfd9858d731101:14214097 l2_safe=0x3a495006a3f2124aee710c4a395ba64af907933b19a1a31955a031b1524c9b28:14214549 l2_pending_safe=0x3a495006a3f2124aee710c4a395ba64af907933b19a1a31955a031b1524c9b28:14214549 l2_unsafe=0xf467918f43f50414718d2cee3c7cb272a53e096a0eea22f551b9552d18cd18ad:14214641 l2_backup_unsafe=0x0000000000000000000000000000000000000000000000000000000000000000:0 l2_time=1715218629 l1_derived=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 node-node-1 | t=2024-05-09T01:37:09+0000 lvl=debug msg="Derivation process step" onto_origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 attempts=0 node-node-1 | t=2024-05-09T01:37:09+0000 lvl=debug msg="can't find next L1 block info (yet)" number=19829249 origin=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 node-node-1 | t=2024-05-09T01:37:09+0000 lvl=debug msg="Derivation process went idle" progress=0x6a557136e2569f92acdcdb8c946f5de5d5b3a83f002a69c7e5776abf091f425b:19829248 err=EOF

with no data and a sync from scratch, here's what I'm getting. Should this actually sync? or do we need to start at a snapshot and wait the 15 hours to head?

[zhy827827]

i have same issue:

ERROR[05-10|05:32:47.226] Failed to create sealing context         err="missing trie node e4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 (path ) state 0xe4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 is not available, not found"
ERROR[05-10|05:32:47.226] Failed to build payload                  err="missing trie node e4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 (path ) state 0xe4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 is not available, not found"
WARN [05-10|05:32:47.226] Served engine_forkchoiceUpdatedV3        reqid=45726 duration=1.58685ms    err="Invalid payload attributes" errdata="{Error:missing trie node e4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 (path ) state 0xe4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 is not available, not found}"

[B1Q]

i have same issue:

ERROR[05-10|05:32:47.226] Failed to create sealing context         err="missing trie node e4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 (path ) state 0xe4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 is not available, not found"
ERROR[05-10|05:32:47.226] Failed to build payload                  err="missing trie node e4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 (path ) state 0xe4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 is not available, not found"
WARN [05-10|05:32:47.226] Served engine_forkchoiceUpdatedV3        reqid=45726 duration=1.58685ms    err="Invalid payload attributes" errdata="{Error:missing trie node e4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 (path ) state 0xe4aa34997d564704f80bff7623f3bbccbd8331f4cafba1edbaecbbd5e698f984 is not available, not found}"

did you find a solution?

[daothanginspius]

I had the same problem today when upgrading to v0.8.3. I tried going back to the old version v0.8.2 but it still didn't work.

[zainirfan13]

facing same issue, was trying to sync and now its stuck on an endless loop.

[rabbitprincess]

t=2024-05-07T22:29:58+0000 lvl=warn msg="Derivation process temporary error" attempts=3911 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x56df86f88d654cfca4a7e72151e61789b84906e8b77a9f720b27ccf3fcc3f000:19521297: failed to fetch blob sidecars for slot 8722070 block 0x56df86f88d654cfca4a7e72151e61789b84906e8b77a9f720b27ccf3fcc3f000:19521297: failed request with status 500: {"code":500,"message":"block not found"}"

Now I am getting these errors, good god.

you need a beacon node which have all historical blobs. https://github.com/ethereum-optimism/optimism/issues/9962

[daothanginspius]

I had the same problem today when upgrading to v0.8.3. I tried going back to the old version v0.8.2 but it still didn't work.

I resolved this issue after using the latest snapshot

[wbnns]

Hello, for people experiencing this issue, could you all please:

1. Confirm you're on 0.8.3 (docker compose stop, git pull to grab latest changes from the repo, and then docker compose up --build)
2. Make sure that in your .env.mainnet or .env.sepolia (depending on the node your running, if on mainnet or testnet) that you aren't in snap sync mode
3. That you're able to reach the L1 and Beacon endpoints provided from the Docker container

If after this the sync isn't progressing and time remaining is not decreasing when running

echo Latest synced block behind by: $((($(date +%s)-$( \
  curl -d '{"id":0,"jsonrpc":"2.0","method":"optimism_syncStatus"}' \
  -H "Content-Type: application/json" http://localhost:7545 | \
  jq -r .result.unsafe_l2.timestamp))/60)) minutes

I would recommend restoring from snapshot as @daothanginspius mentioned.

If anyone would like individualize support on this, please feel free to come to Base Discord and open a #developer-support ticket. We are happy to try to help you there. 👍

[wbnns]

Since it has been a couple of weeks without further updates, I'm going ahead and closing this issue. If anyone is still encountering a problem, please don't hesitate to stop by the Base Discord and post in the #node-operators channel. We can help you there! :)

[usoftvn]

I had the same problem today when upgrading to v0.8.3. I tried going back to the old version v0.8.2 but it still didn't work.

I fixed it by using the new snapshot. Thank you!

[zhy827827]

I solve this issues! The root cause is the problem of data corruption