optimism
optimism copied to clipboard
Published
20 hours ago •
ethereum-optimism
ethereum-optimism
L2geth Regenerated local transaction journal
Describe the bug L2geth stop synchronization process.
To Reproduce start.sh:
export USING_OVM=true
export SEQUENCER_CLIENT_HTTP=https://mainnet.optimism.io
export BLOCK_SIGNER_ADDRESS=0x00000398232E2064F896018496b4b44b3D62751F
export BLOCK_SIGNER_PRIVATE_KEY=6587ae678cf4fc9a33000cdbf9f35226b71dcc6a4684a31203241f9bcfd55d27
#export BLOCK_SIGNER_PRIVATE_KEY_PASSWORD=pwd
export ETH1_CTC_DEPLOYMENT_HEIGHT=13596466
export ETH1_SYNC_SERVICE_ENABLE=true
export L2GETH_GENESIS_URL=https://storage.googleapis.com/optimism/mainnet/genesis-berlin.json
export L2GETH_GENESIS_HASH=0x106b0a3247ca54714381b1109e82cc6b7e32fd79ae56fbcc2e7b1541122f84ea
export L2GETH_BERLIN_ACTIVATION_HEIGHT=3950000
export ROLLUP_BACKEND=l1
export ROLLUP_CLIENT_HTTP=http://0.0.0.0:7878
export ROLLUP_DISABLE_TRANSFERS=false
export ROLLUP_ENABLE_L2_GAS_POLLING=false
export ROLLUP_ADDRESS_MANAGER_OWNER_ADDRESS=0x9BA6e03D8B90dE867373Db8cF1A58d2F7F006b3A
export ROLLUP_GAS_PRICE_ORACLE_OWNER_ADDRESS=0x648E3e8101BFaB7bf5997Bd007Fb473786019159
export ROLLUP_MAX_CALLDATA_SIZE=40000
export ROLLUP_POLL_INTERVAL_FLAG=1s
export ROLLUP_SYNC_SERVICE_ENABLE=true
export ROLLUP_TIMESTAMP_REFRESH=5m
export ROLLUP_VERIFIER_ENABLE=true
export GCMODE=archive
export DATADIR=/home/opt/l2geth/chaindata
export CHAIN_ID=10
export NETWORK_ID=10
export NO_DISCOVER=true
export NO_USB=true
export GASPRICE=0
export TARGET_GAS_LIMIT=15000000
export RPC_ADDR=0.0.0.0
export RPC_API=eth,rollup,net,web3,debug
export RPC_CORS_DOMAIN=*
export RPC_ENABLE=true
export RPC_PORT=28545
export RPC_VHOSTS=*
export WS_ADDR=0.0.0.0
export WS_API=eth,rollup,net,web3,debug
export WS_ORIGINS=*
export WS=true
export L1_HTTP_API_URL=http://0.0.0.0:8546
./geth \
--datadir.ancient="./ancient" \
--datadir=$DATADIR \
--password=$DATADIR/password \
--allow-insecure-unlock \
--unlock=$BLOCK_SIGNER_ADDRESS \
--mine \
--miner.etherbase=$BLOCK_SIGNER_ADDRESS \
--wsport=28546
Expected behavior
INFO [02-01|09:18:14.940] Maximum peer count ETH=50 LES=0 total=50
INFO [02-01|09:18:14.940] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [02-01|09:18:14.940] Starting peer-to-peer node instance=Geth/v1.9.10-stable/linux-amd64/go1.19
INFO [02-01|09:18:14.940] Allocated trie memory caches clean=512.00MiB dirty=0.00B
INFO [02-01|09:18:14.940] Allocated cache and file handles database=/home/opt/optimism-mainnet/l2geth/chaindata/geth/chaindata cache=512.00MiB handles=524288
INFO [02-01|09:20:26.281] Opened ancient database database=/home/opt/optimism-mainnet/l2geth/chaindata/geth/ancient
INFO [02-01|09:20:26.330] Initialised chain configuration config="{ChainID: 10 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Berli>
INFO [02-01|09:20:26.331] Initialising Ethereum protocol versions="[64 63]" network=10 dbversion=7
INFO [02-01|09:20:26.335] Loaded most recent local header number=42877317 hash=66f24d…2f8426 td=85754635 age=2mo5d19h
INFO [02-01|09:20:26.335] Loaded most recent local full block number=42877317 hash=66f24d…2f8426 td=85754635 age=2mo5d19h
INFO [02-01|09:20:26.335] Loaded most recent local fast block number=42877317 hash=66f24d…2f8426 td=85754635 age=2mo5d19h
INFO [02-01|09:20:26.405] Loaded local transaction journal transactions=0 dropped=0
INFO [02-01|09:20:26.405] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|09:20:26.405] Running in verifier mode sync-backend=l1
INFO [02-01|09:20:26.405] Configured rollup client url=http://0.0.0.0:7878 chain-id=10 ctc-deploy-height=13596466
INFO [02-01|09:20:26.751] Connected to upstream service
INFO [02-01|09:20:26.758] Updating batch index old=0xc009ba9b38 new=318978
INFO [02-01|09:20:26.759] Found latest index index=42877317
INFO [02-01|09:20:26.760] Block not found, resetting index new=42877316 old=42877317
INFO [02-01|09:20:26.761] Found latest queue index queue-index=216768
INFO [02-01|09:20:26.827] Found correct staring queue index queue-index=216768
INFO [02-01|09:20:26.828] Initialized Latest L1 Info blocknumber=16061797 timestamp=1669558419
INFO [02-01|09:20:26.828] Initial Rollup State state=0x15436912d15aaec2a841595493d83113468e81676e27a98245fd4540960a3496 index=42877316 queue-index=216768 verified-index=0xc00a9b7b00
WARN [02-01|09:20:26.828] Switch sync mode from fast sync to full sync
INFO [02-01|09:20:26.828] Backend Config max-calldata-size=40000 gas-limit=15000000 is-verifier=true using-ovm=true
INFO [02-01|09:20:26.832] New local node record seq=3 id=b684181023cb9d87 ip=127.0.0.1 udp=0 tcp=30303
INFO [02-01|09:20:26.832] Started P2P networking self="enode://e6e5c9244a1d3219868079694f771511e263954003163603daa749713bf6fcd14ff9c49e79b922fb08939c22aa795826298f33121612765f998a4142bebe2a4b@127.0.0.1:30303?discport=0"
INFO [02-01|09:20:26.832] IPC endpoint opened url=/home/opt/optimism-mainnet/l2geth/chaindata/geth.ipc
INFO [02-01|09:20:26.833] HTTP endpoint opened url=http://0.0.0.0:28545 cors=* vhosts=*
INFO [02-01|09:20:26.833] WebSocket endpoint opened url=ws://[::]:28546
INFO [02-01|09:20:27.323] Unlocked account address=0x00000398232E2064F896018496b4b44b3D62751F
INFO [02-01|09:20:27.323] Transaction pool price threshold updated price=0
INFO [02-01|09:20:27.323] Transaction pool price threshold updated price=0
INFO [02-01|09:20:27.323] Initializing Sync Service
INFO [02-01|09:20:27.324] Set L2 Gas Price gasprice=1000000
INFO [02-01|09:20:27.324] Set L1 Gas Price gasprice=8806392208
INFO [02-01|09:20:27.324] Set batch overhead overhead=2100
INFO [02-01|09:20:27.324] Set scalar scalar=1
INFO [02-01|09:20:27.324] Starting Verifier Loop poll-interval=1s timestamp-refresh-threshold=5m0s
INFO [02-01|09:20:27.410] Syncing transaction batch range start=318979 end=509378
INFO [02-01|09:20:27.797] New block index=42877317 l1-timestamp=1669558419 l1-blocknumber=16061797 tx-hash=0xe9bdb3512054aa7d1682da3bac7668c33057767bb815f4c7061f27faed020eaf queue-orign=sequencer gas=398775 fees=3.98775e-07 elapsed=141.076ms
INFO [02-01|10:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|11:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|12:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|13:20:26.411] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|14:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|15:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|16:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|17:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|18:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|19:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|20:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|21:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|22:20:26.407] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-01|23:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|00:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|01:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|02:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|03:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|04:20:26.411] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|05:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|06:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|07:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|08:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|09:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|10:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-02|11:20:26.406] Regenerated local transaction journal transactions=0 accounts=0
System Specs:
5.15.0-57-generic #63-Ubuntu SMP Thu Nov 24 13:43:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Usually indicates an issue with the DTL. Can you check your DTL logs? Double check that your connection to the L1 geth node is OK.
Likely a duplicate of: https://github.com/ethereum-optimism/optimism/issues/4754, am working on a fix
@smartcontracts could you please create a branch or tag at github. I will download and check.
@artjoma heres the branch: https://github.com/ethereum-optimism/optimism/tree/sc/l2g-deadlock
Time to time starting long running LevelDB compaction:
INFO [02-10|09:49:14.215] New block index=46937552 l1-timestamp=1670547788 l1-blocknumber=16143694 tx-hash=0x0d8d2cdce0e7f9269ae01623b6599fc5211f07f329d46413daacb7892635242e queue-orign=sequencer gas=827498 fees=8.27498e-07 elapsed=69.024ms
INFO [02-10|09:49:14.227] New block index=46937553 l1-timestamp=1670547788 l1-blocknumber=16143694 tx-hash=0xd01bb9669c471f2c04d3f9b8e0833dfd2d36267dd9ee202fe1251773d424aaf3 queue-orign=sequencer gas=179696 fees=1.79696e-07 elapsed=2.186ms
INFO [02-10|09:49:14.238] New block index=46937554 l1-timestamp=1670547788 l1-blocknumber=16143694 tx-hash=0xca752732a51a0a26095f0a5e438bdfc217a7ae3f647d1a6132951148f99227fe queue-orign=sequencer gas=216632 fees=2.16632e-07 elapsed=2.696ms
INFO [02-10|09:49:14.247] New block index=46937555 l1-timestamp=1670547788 l1-blocknumber=16143694 tx-hash=0xbef72bd0058df1a99314df1bd49b8259c71f582c45864aa01ebee71cc4ae7b8d queue-orign=sequencer gas=21000 fees=2.1e-08 elapsed=715.925µs
WARN [02-10|09:49:14.815] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:50:15.143] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:51:15.451] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:52:15.763] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:53:16.049] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:54:16.299] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:55:16.567] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:56:16.826] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:57:17.129] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:58:17.425] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|09:59:17.740] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:00:18.048] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:01:18.324] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:02:18.632] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
...
INFO [02-10|10:24:33.616] New block index=46982868 l1-timestamp=1670558225 l1-blocknumber=16144562 tx-hash=0x7fc54579819145d908776f85049c7c80918b25798e04cb9c9d23782c572d40a6 queue-orign=sequencer gas=102501 fees=1.02501e-07 elapsed=5.753ms
WARN [02-10|10:24:34.033] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:25:34.320] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:26:34.650] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:27:34.950] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:28:35.226] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:29:35.533] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:30:35.788] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:31:36.148] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:32:36.453] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:33:36.750] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:34:37.047] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:35:37.343] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:36:37.626] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:37:37.918] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:38:38.178] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:39:38.507] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:40:38.788] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-10|10:41:39.086] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:35:06.397] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:36:06.680] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:37:06.962] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:38:07.263] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:39:07.562] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:40:07.807] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:41:08.092] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:42:08.347] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:43:08.620] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:44:08.878] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:45:09.138] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:46:09.392] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:47:09.658] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:48:09.932] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:49:10.191] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:50:10.473] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:51:10.721] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:52:10.986] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:53:11.270] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:54:11.557] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:55:11.839] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:56:12.093] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:57:12.374] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
WARN [02-11|12:58:12.649] Database compacting, degraded performance database=/home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata
ERROR[02-11|12:58:14.175] Failed to write trie to disk err="newManifest error: write /home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata/MANIFEST-22860444: no space left on device, cleanup error (<nil>)"
ERROR[02-11|12:58:14.175] Failed writing block to chain err="newManifest error: write /home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata/MANIFEST-22860444: no space left on device, cleanup error (<nil>)"
CRIT [02-11|12:58:14.176] Failed to delete frozen canonical blocks err="newManifest error: write /home/blockchain/optimism-mainnet/l2geth/chaindata/geth/chaindata/MANIFEST-22860444: no space left on device, cleanup error (<nil>)"
/home/blockchain/optimism-mainnet/l2geth# du -hs chaindata/
9.5T chaindata/