optimism insufficient balance for transfer

Describe the bug Not synchronize

To Reproduce Steps to reproduce the behavior:

start l2geth
infinity write to log insufficient balance for transfer

Screenshots

INFO [10-12|08:16:22.432] Starting peer-to-peer node               instance=Geth/v1.9.10-stable/linux-amd64/go1.19
INFO [10-12|08:16:22.433] Allocated trie memory caches             clean=512.00MiB dirty=0.00B
INFO [10-12|08:16:22.433] Allocated cache and file handles         database=/mnt/md2/optimism-mainnet/l2geth/chaindata/geth/chaindata cache=512.00MiB handles=350000
INFO [10-12|08:16:25.988] Opened ancient database                  database=/mnt/md2/optimism-mainnet/l2geth/chaindata/geth/chaindata/ancient
INFO [10-12|08:16:25.991] Initialised chain configuration          config="{ChainID: 10 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Berlin: 3950000, Engine: clique}"
INFO [10-12|08:16:25.991] Initialising Ethereum protocol           versions="[64 63]" network=10 dbversion=7
INFO [10-12|08:16:26.024] Loaded most recent local header          number=22563164 hash=d1d449…394814 td=45126329 age=1mo2d9h
INFO [10-12|08:16:26.024] Loaded most recent local full block      number=22563164 hash=d1d449…394814 td=45126329 age=1mo2d9h
INFO [10-12|08:16:26.024] Loaded most recent local fast block      number=22563164 hash=d1d449…394814 td=45126329 age=1mo2d9h
INFO [10-12|08:16:26.067] Loaded local transaction journal         transactions=0 dropped=0
INFO [10-12|08:16:26.067] Regenerated local transaction journal    transactions=0 accounts=0
INFO [10-12|08:16:26.067] Running in verifier mode                 sync-backend=l1
INFO [10-12|08:16:26.067] Configured rollup client                 url=http://0.0.0.0:7878 chain-id=10 ctc-deploy-height=13596466
INFO [10-12|08:16:26.081] Connected to upstream service 
INFO [10-12|08:16:26.084] Updating batch index                     old=0xc002398958 new=135659
INFO [10-12|08:16:26.084] Found latest index                       index=22563163
INFO [10-12|08:16:26.084] Found latest queue index                 queue-index=141610
INFO [10-12|08:16:26.107] Found correct staring queue index        queue-index=141610
INFO [10-12|08:16:26.107] Initialized Latest L1 Info               blocknumber=15505238 timestamp=1662762862
INFO [10-12|08:16:26.107] Initial Rollup State                     state=0x179da627f837b6ab7e27ec76cfb781b3620a2dd6d957ee4ca4b7637d26cee8f5 index=22563163 queue-index=141610 verified-index=0xc00c115398
WARN [10-12|08:16:26.107] Switch sync mode from fast sync to full sync 
INFO [10-12|08:16:26.107] Backend Config                           max-calldata-size=400 gas-limit=500000000 is-verifier=true using-ovm=true
INFO [10-12|08:16:26.112] New local node record                    seq=34 id=48fe0e4c88a169d0 ip=127.0.0.1 udp=0 tcp=30303
INFO [10-12|08:16:26.112] Started P2P networking                   self="enode://e46b2c7e6a916d9460ce425e177d92bf43c12a0e38d9528059723f55f10ef1bef28ad58936a4f825064660ef768427fede936d183cc13258360db1f190cd18fe@127.0.0.1:30303?discport=0"
INFO [10-12|08:16:26.113] IPC endpoint opened                      url=/mnt/md2/optimism-mainnet/l2geth/chaindata/geth.ipc
INFO [10-12|08:16:26.114] HTTP endpoint opened                     url=http://0.0.0.0:28545 cors=* vhosts=*
INFO [10-12|08:16:26.114] WebSocket endpoint opened                url=ws://[::]:8546
INFO [10-12|08:16:33.638] Unlocked account                         address=0x00000398232E2064F896018496b4b44b3D62751F
INFO [10-12|08:16:33.638] Transaction pool price threshold updated price=0
INFO [10-12|08:16:33.638] Transaction pool price threshold updated price=0
INFO [10-12|08:16:33.638] Initializing Sync Service 
INFO [10-12|08:16:33.644] Set L2 Gas Price                         gasprice=1000000
INFO [10-12|08:16:33.644] Set L1 Gas Price                         gasprice=14663696100
INFO [10-12|08:16:33.645] Set batch overhead                       overhead=2100
INFO [10-12|08:16:33.647] Set scalar                               scalar=1
INFO [10-12|08:16:33.647] Starting Verifier Loop                   poll-interval=1s timestamp-refresh-threshold=5m0s
INFO [10-12|08:16:33.661] Syncing transaction batch range          start=135660 end=192229
ERROR[10-12|08:16:33.705] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[10-12|08:16:33.705] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[10-12|08:16:33.705] Could not verify                         error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction: Cannot apply batched transaction: insufficient balance for transfer"
INFO [10-12|08:16:34.662] Syncing transaction batch range          start=135660 end=192229
ERROR[10-12|08:16:34.683] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[10-12|08:16:34.683] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[10-12|08:16:34.683] Could not verify                         error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction: Cannot apply batched transaction: insufficient balance for transfer"
INFO [10-12|08:16:35.663] Syncing transaction batch range          start=135660 end=192229
ERROR[10-12|08:16:35.690] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[10-12|08:16:35.690] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[10-12|08:16:35.690] Could not verify                         error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction: Cannot apply batched transaction: insufficient balance for transfer"
INFO [10-12|08:16:36.658] Syncing transaction batch range          start=135660 end=192229
ERROR[10-12|08:16:36.683] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[10-12|08:16:36.683] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[10-12|08:16:36.683] Could not verify                         error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction: Cannot apply batched transaction: insufficient balance for transfer"
^CINFO [10-12|08:16:37.019] Got interrupt, shutting down...

System Specs:

OS: Linux
0.5.24 (Geth/v1.9.10-stable)

Additional context

export USING_OVM=true
export SEQUENCER_CLIENT_HTTP=https://mainnet.optimism.io
export BLOCK_SIGNER_ADDRESS=0x00000398232E2064F896018496b4b44b3D62751F
export BLOCK_SIGNER_PRIVATE_KEY=6587ae678cf4fc9a33000cdbf9f35226b71dcc6a4684a31203241f9bcfd55d27
export ETH1_CTC_DEPLOYMENT_HEIGHT=13596466
export ETH1_SYNC_SERVICE_ENABLE=true
export L2GETH_GENESIS_URL=https://storage.googleapis.com/optimism/mainnet/genesis-berlin.json
export L2GETH_GENESIS_HASH=0x106b0a3247ca54714381b1109e82cc6b7e32fd79ae56fbcc2e7b1541122f84ea
export L2GETH_BERLIN_ACTIVATION_HEIGHT=3950000
export ROLLUP_BACKEND=l1
export ROLLUP_CLIENT_HTTP=http://0.0.0.0:7878
export ROLLUP_DISABLE_TRANSFERS=false
export ROLLUP_ENABLE_L2_GAS_POLLING=false
export ROLLUP_ADDRESS_MANAGER_OWNER_ADDRESS=0x9BA6e03D8B90dE867373Db8cF1A58d2F7F006b3A
export ROLLUP_GAS_PRICE_ORACLE_OWNER_ADDRESS=0x648E3e8101BFaB7bf5997Bd007Fb473786019159
export ROLLUP_MAX_CALLDATA_SIZE=40000
export ROLLUP_POLL_INTERVAL_FLAG=1s
export ROLLUP_SYNC_SERVICE_ENABLE=true
export ROLLUP_TIMESTAMP_REFRESH=5m
export ROLLUP_VERIFIER_ENABLE=true

export GCMODE=archive
export DATADIR=/mnt/md2/optimism-mainnet/l2geth/chaindata
export CHAIN_ID=10
export NETWORK_ID=10
export NO_DISCOVER=true
export NO_USB=true
export GASPRICE=0
export TARGET_GAS_LIMIT=500000000
export RPC_ADDR=0.0.0.0
export RPC_API=eth,rollup,net,web3,debug
export RPC_CORS_DOMAIN=*
export RPC_ENABLE=true
export RPC_PORT=28545
export RPC_VHOSTS=*
export WS_ADDR=0.0.0.0
export WS_API=eth,rollup,net,web3,debug
export WS_ORIGINS=*
export WS=true


./geth \
   --datadir=$DATADIR \
   --password=$DATADIR/password \
   --allow-insecure-unlock \
   --unlock=$BLOCK_SIGNER_ADDRESS \
   --mine \
   --miner.etherbase=$BLOCK_SIGNER_ADDRESS \

Oct 12 '22 08:10 artjoma

++

Oct 12 '22 08:10 jacquesvcritien

++

Oct 12 '22 08:10 franono

++

Oct 14 '22 04:10 yj0x0x

Hmm... people ++'ing this, are you getting this result on the same block that the original poster is?

Oct 18 '22 05:10 smartcontracts

AKA INFO [10-12|08:16:33.661] Syncing transaction batch range start=135660 end=192229

Oct 18 '22 05:10 smartcontracts

And are you syncing from L1 or L2 (DTL config).

Oct 18 '22 05:10 smartcontracts

@smartcontracts from L1.

DTL config:

DATA_TRANSPORT_LAYER__ADDRESS_MANAGER=0xdE1FCfB0851916CA5101820A69b13a4E276bd81F
DATA_TRANSPORT_LAYER__SYNC_FROM_L1=true
DATA_TRANSPORT_LAYER__SYNC_FROM_L2=false
DATA_TRANSPORT_LAYER__L1_START_HEIGHT=13596466
DATA_TRANSPORT_LAYER__CONFIRMATIONS=12
DATA_TRANSPORT_LAYER__DANGEROUSLY_CATCH_ALL_ERRORS=true
DATA_TRANSPORT_LAYER__DB_PATH=/db
DATA_TRANSPORT_LAYER__DEFAULT_BACKEND=l1
DATA_TRANSPORT_LAYER__L1_GAS_PRICE_BACKEND=l1
DATA_TRANSPORT_LAYER__ENABLE_METRICS=true
DATA_TRANSPORT_LAYER__ETH_NETWORK_NAME=mainnet
DATA_TRANSPORT_LAYER__L2_CHAIN_ID=10
DATA_TRANSPORT_LAYER__LOGS_PER_POLLING_INTERVAL=2000
DATA_TRANSPORT_LAYER__NODE_ENV=production
DATA_TRANSPORT_LAYER__POLLING_INTERVAL=500
DATA_TRANSPORT_LAYER__SENTRY_TRACE_RATE=0.05
DATA_TRANSPORT_LAYER__SERVER_HOSTNAME=0.0.0.0
DATA_TRANSPORT_LAYER__SERVER_PORT=7878
DATA_TRANSPORT_LAYER__TRANSACTIONS_PER_POLLING_INTERVAL=1000
DATA_TRANSPORT_LAYER__L1_RPC_ENDPOINT=http://0.0.0.0:11101

Oct 18 '22 07:10 artjoma

Can everyone here confirm that they're using a target gas limit of exactly 15000000?

Oct 18 '22 19:10 smartcontracts

Target gas limit must be set to exactly 15000000

Oct 18 '22 19:10 smartcontracts

@smartcontracts thanks for the follow up. Replying in behalf of duplicate issue 3646, that parameter is configured as mentioned:

L2GETH runner (same envs config)
Environment="TARGET_GAS_LIMIT=15000000"

I recommend taking a look to that other issue as it can have complimentary data for the investigation.

Oct 18 '22 22:10 runatyr1

@smartcontracts I tried with multiply values: export TARGET_GAS_LIMIT=20000000, 15000000, 16000000 and 500000000. Anyway I got error: ERROR[10-19|09:13:44.892] Could not verify error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction:

Oct 19 '22 09:10 artjoma

the solution is using

export USING_OVM=true

before initializing L2geth (l2geth init)

Oct 20 '22 00:10 gabronick

Ah yep, definitely need USING_OVM when initializing L2Geth.

Oct 20 '22 19:10 smartcontracts

I initialized chain again using USING_OVM=true:

INFO [10-23|17:39:25.493] Set L1 Gas Price                         gasprice=149453128946
INFO [10-23|17:39:25.493] Set batch overhead                       overhead=2750
INFO [10-23|17:39:25.493] Set scalar                               scalar=1.5
INFO [10-23|17:39:25.494] New block                                index=203   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0xd82a58f05db58a65446438ea6ac20b2ba1f0387e8342862f8e82b820adbc51c6 queue-orign=sequencer gas=29748   fees=0            elapsed=1.024ms
INFO [10-23|17:39:25.496] Set L2 Gas Price                         gasprice=1000000
INFO [10-23|17:39:25.496] Set L1 Gas Price                         gasprice=126492216602
INFO [10-23|17:39:25.496] Set batch overhead                       overhead=2750
INFO [10-23|17:39:25.496] Set scalar                               scalar=1.5
INFO [10-23|17:39:25.497] New block                                index=204   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0x81d50f654048cc66e56de021845e7f41d5322a2bd4e10885f69bb584708e83f8 queue-orign=sequencer gas=21000   fees=2.1e-08      elapsed=1.581ms
INFO [10-23|17:39:25.500] New block                                index=205   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0xe124c637d83ee40742f74e51c643dd9b2f57b0d7606326b0d4f3ce0163688217 queue-orign=sequencer gas=570074  fees=5.70074e-07  elapsed=1.495ms
INFO [10-23|17:39:25.503] New block                                index=206   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0x2ed772cd7faa939905d920608d72133f6b5f74e6bb889cd055076c653d66a2c2 queue-orign=sequencer gas=44006   fees=4.4006e-08   elapsed=1.081ms
INFO [10-23|17:39:25.521] New block                                index=207   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0x3bf61ac5c9debe380401d2fc06124335d80b5cdd986276d6094b033d427a4e3e queue-orign=sequencer gas=55666   fees=5.5666e-08   elapsed=15.585ms
INFO [10-23|17:39:25.523] New block                                index=208   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0x1aa3ed5463e36070930b8b72eace4a1e73ad62bd5b4910fb8ea25835b57963c1 queue-orign=sequencer gas=14417   fees=1.4417e-08   elapsed=1.089ms
INFO [10-23|17:39:25.526] New block                                index=209   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0x87dd43f032f033a577bd7a8ca5f0076c139f74d97b6f8b011830d658ae274517 queue-orign=sequencer gas=570038  fees=5.70038e-07  elapsed=1.542ms
INFO [10-23|17:39:25.529] New block                                index=210   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0xfc35df5ed7d5429cf286c69ef64107a4a2c6e8e41fab47014c427046eac63bc7 queue-orign=sequencer gas=570038  fees=5.70038e-07  elapsed=1.391ms
INFO [10-23|17:39:25.531] New block                                index=211   l1-timestamp=1636670191 l1-blocknumber=13597603 tx-hash=0x3cc7b63d6367639bc5e1a564f6a2f59e215d2b4d55b36180e0df03a28c204d6a queue-orign=sequencer gas=29748   fees=0            elapsed=836.698µs
INFO [10-23|17:39:25.531] Set L2 Gas Price                         gasprice=1000000
INFO [10-23|17:39:25.532] Set L1 Gas Price                         gasprice=160078972504
INFO [10-23|17:39:25.532] Set batch overhead                       overhead=2750

After few hours:

INFO [10-23|17:29:31.161] Syncing transaction batch range          start=15416 end=218082
ERROR[10-23|17:29:31.172] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[10-23|17:29:31.172] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[10-23|17:29:31.172] Could not verify                         error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction: Cannot apply batched transaction: insufficient balance for transfer"
INFO [10-23|17:29:32.163] Syncing transaction batch range          start=15416 end=218082
ERROR[10-23|17:29:32.173] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[10-23|17:29:32.173] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[10-23|17:29:32.173] Could not verify                         error="Verifier cannot sync transaction batches to tip: Cannot sync transaction batches to tip: Cannot sync batches: cannot apply batched transaction: Cannot apply batched transaction: insufficient balance for transfer"

May be documentation not up to date: https://community.optimism.io/docs/developers/build/run-a-node/#the-optimism-client-software ?

Oct 23 '22 17:10 artjoma

@artjoma can you confirm you're using TARGET_GAS_LIMIT=15000000?

Oct 25 '22 19:10 smartcontracts

@smartcontracts export TARGET_GAS_LIMIT=16000000

Oct 26 '22 08:10 artjoma

Please try with export TARGET_GAS_LIMIT=15000000

Oct 26 '22 15:10 smartcontracts

same issue, different block

ERROR[11-04|18:15:12.909] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[11-04|18:15:12.909] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[11-04|18:15:12.909] Could not verify                         error="Verifier cannot sync transactions with BackendL2: Verifier cannot sync transactions with backend l2: Cannot sync transactions with backend l2: Cannot apply transaction: insufficient balance for transfer"

tried all mentioned solutions.

Nov 04 '22 18:11 unek

same issue, different block

ERROR[11-04|18:15:12.909] Problem committing transaction           msg="insufficient balance for transfer"
ERROR[11-04|18:15:12.909] Got error waiting for transaction to be added to chain msg="insufficient balance for transfer"
ERROR[11-04|18:15:12.909] Could not verify                         error="Verifier cannot sync transactions with BackendL2: Verifier cannot sync transactions with backend l2: Cannot sync transactions with backend l2: Cannot apply transaction: insufficient balance for transfer"

tried all mentioned solutions.

Note that you'll need to resync the chain if you change the block gas limit.

Nov 04 '22 23:11 smartcontracts

Note that you'll need to resync the chain if you change the block gas limit.

I did not change it

Nov 05 '22 12:11 unek

@unek is your block gas limit 15m?

Nov 06 '22 22:11 smartcontracts

@unek is your block gas limit 15m?

yes, it is, set via export TARGET_GAS_LIMIT=15000000. as I said, I tried all solutions mentioned in this thread.

Nov 06 '22 22:11 unek

resynced l2geth, nothing's helping.

Nov 07 '22 02:11 unek

@smartcontracts I solved this issue using TARGET_GAS_LIMIT=15000000 and USING_OVM=true

@unek

init.sh :

export USING_OVM=true
export DATADIR=/mnt/md1/optimism-mainnet/l2geth/chaindata
export BLOCK_SIGNER_PRIVATE_KEY=6587ae678cf4fc9a33000cdbf9f35226b71dcc6a4684a31203241f9bcfd55d27

./geth init --datadir=$DATADIR genesis-berlin.json --nousb
touch $DATADIR/password
echo $BLOCK_SIGNER_PRIVATE_KEY > $DATADIR/block-signer-key
./geth account import --datadir=$DATADIR --password $DATADIR/password $DATADIR/block-signer-key

start.sh:

export USING_OVM=true
export SEQUENCER_CLIENT_HTTP=https://mainnet.optimism.io
export BLOCK_SIGNER_ADDRESS=0x00000398232E2064F896018496b4b44b3D62751F
export BLOCK_SIGNER_PRIVATE_KEY=6587ae678cf4fc9a33000cdbf9f35226b71dcc6a4684a31203241f9bcfd55d27
export ETH1_CTC_DEPLOYMENT_HEIGHT=13596466
export ETH1_SYNC_SERVICE_ENABLE=true
export L2GETH_GENESIS_URL=https://storage.googleapis.com/optimism/mainnet/genesis-berlin.json
export L2GETH_GENESIS_HASH=0x106b0a3247ca54714381b1109e82cc6b7e32fd79ae56fbcc2e7b1541122f84ea
export L2GETH_BERLIN_ACTIVATION_HEIGHT=3950000
export ROLLUP_BACKEND=l1
export ROLLUP_CLIENT_HTTP=http://0.0.0.0:7878
export ROLLUP_DISABLE_TRANSFERS=false
export ROLLUP_ENABLE_L2_GAS_POLLING=false
export ROLLUP_ADDRESS_MANAGER_OWNER_ADDRESS=0x9BA6e03D8B90dE867373Db8cF1A58d2F7F006b3A
export ROLLUP_GAS_PRICE_ORACLE_OWNER_ADDRESS=0x648E3e8101BFaB7bf5997Bd007Fb473786019159
export ROLLUP_MAX_CALLDATA_SIZE=40000
export ROLLUP_POLL_INTERVAL_FLAG=1s
export ROLLUP_SYNC_SERVICE_ENABLE=true
export ROLLUP_TIMESTAMP_REFRESH=5m
export ROLLUP_VERIFIER_ENABLE=true

export GCMODE=archive
export DATADIR=/mnt/md1/optimism-mainnet/l2geth/chaindata
export CHAIN_ID=10
export NETWORK_ID=10
export NO_DISCOVER=true
export NO_USB=true
export GASPRICE=0
export TARGET_GAS_LIMIT=15000000
export RPC_ADDR=0.0.0.0
export RPC_API=eth,rollup,net,web3,debug
export RPC_CORS_DOMAIN=*
export RPC_ENABLE=true
export RPC_PORT=28545
export RPC_VHOSTS=*
export WS_ADDR=0.0.0.0
export WS_API=eth,rollup,net,web3,debug
export WS_ORIGINS=*
export WS=true

./geth \
   --datadir.ancient="./ancient" \
   --datadir=$DATADIR \
   --password=$DATADIR/password \
   --allow-insecure-unlock \
   --unlock=$BLOCK_SIGNER_ADDRESS \
   --mine \
   --miner.etherbase=$BLOCK_SIGNER_ADDRESS \

Right now chaindata size: 1T

Nov 07 '22 13:11 artjoma

@artjoma you can use GCMODE=full if you don't need the entire archive state. Should be much smaller.

Nov 15 '22 17:11 smartcontracts

@smartcontracts I need archive node with all state history. Thank you!

Nov 16 '22 07:11 artjoma

optimism optimism copied to clipboard

insufficient balance for transfer

optimism
optimism copied to clipboard