optimism
optimism copied to clipboard
Published
20 hours ago •
ethereum-optimism
ethereum-optimism
Batch submitter crash on local network
Describe the bug Whenever using the local network from docker-compose as: https://github.com/ethereum-optimism/optimism/tree/develop/ops/ After submitting a transaction the batch_submitter hard crash with:
INFO [06-01|04:27:54.877] Sequencer updating batch tx gas price start=1 end=25 nonce=1
INFO [06-01|04:27:54.890] Proposer fetching current block range
WARN [06-01|04:27:54.892] Sequencer eth_maxPriorityFeePerGas is unsupported by current backend, using fallback gasTipCap
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x513234]
goroutine 63 [running]:
math/big.(*Int).Mul(0x40002a7968, 0x0, 0x40002a7988)
/usr/local/go/src/math/big/int.go:168 +0xa4
github.com/ethereum-optimism/optimism/bss-core/txmgr.CalcGasFeeCap(0x4000010060?, 0xd18fa8?)
/go/bss-core/txmgr/txmgr.go:335 +0x68
github.com/ethereum-optimism/optimism/batch-submitter/drivers/sequencer.(*Driver).UpdateGasPrice(0x4000454a50, {0xd18fa8?, 0x4000260fc0}, 0x40004a4d80)
/go/batch-submitter/drivers/sequencer/driver.go:340 +0xf8
github.com/ethereum-optimism/optimism/bss-core.(*Service).eventLoop.func1({0xd18fa8, 0x4000260fc0})
/go/bss-core/service.go:209 +0x138
github.com/ethereum-optimism/optimism/bss-core/txmgr.(*SimpleTxManager).Send.func1()
/go/bss-core/txmgr/txmgr.go:141 +0xac
created by github.com/ethereum-optimism/optimism/bss-core/txmgr.(*SimpleTxManager).Send
/go/bss-core/txmgr/txmgr.go:205 +0x260
To Reproduce Steps to reproduce the behavior:
- Clone https://github.com/ethereum-optimism/optimism/tree/develop/ops/
- Go to ops folder, docker-compose pull
- docker-compose up
- Wait for the system to get stable
- Submit a transaction on l2
- Check batch_submitter logs or start it again with
docker-compose up batch_submitter - See it crashing
Expected behavior Batch submitter should submit batch to L1 without crashing
System Specs:
- OS: MAC 12.04
- Package Version (or commit hash): master or develop as both would use the "latest" available image
Additional context I have tried to recompile the batch_submitter with a hardcoded fee of 0x1 with something like:
--- a/batch-submitter/drivers/sequencer/driver.go
+++ b/batch-submitter/drivers/sequencer/driver.go
@@ -333,11 +333,11 @@ func (d *Driver) UpdateGasPrice(
gasTipCap = drivers.FallbackGasTipCap
}
- header, err := d.cfg.L1Client.HeaderByNumber(ctx, nil)
- if err != nil {
- return nil, err
- }
- gasFeeCap := txmgr.CalcGasFeeCap(header.BaseFee, gasTipCap)
+ //header, err := d.cfg.L1Client.HeaderByNumber(ctx, nil)
+ //if err != nil {
+ // return nil, err
+ //}
+ gasFeeCap := txmgr.CalcGasFeeCap(big.NewInt(1), gasTipCap)
After that it doesn't crash and I started seeing something like:
batch_submitter_1 | ERROR[06-01|14:28:47.257] Sequencer unable to update txn gas price err="maxFeePerGas or maxPriorityFeePerGas specified but london is not active yet"
What are you using as L1? The L1 node provided in the docker-compose setup?
Yes @smartcontracts , I just pruned my whole docker (docker system prune -a) to be sure and I'm getting the same result.
optimism % git pull origin master
From https://github.com/ethereum-optimism/optimism
* branch master -> FETCH_HEAD
Already up to date.
optimism % cd ops
ops % docker-compose pull
Pulling l1_chain ... done
Pulling deployer ... done
Pulling l2geth ... done
Pulling dtl ... done
Pulling relayer ... done
Pulling fault_detector ... done
Pulling verifier ... done
Pulling replica ... done
Pulling replica_healthcheck ... done
Pulling integration_tests ... done
Pulling gas_oracle ... done
Pulling batch_submitter ... done
ops % docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ethereumoptimism/gas-oracle latest 50cb2ad84f1e 2 minutes ago 21.8MB
ethereumoptimism/message-relayer latest 8f1e7fc7343d 2 weeks ago 1.28GB
ethereumoptimism/batch-submitter-service latest 76b1d7b32404 2 weeks ago 28.9MB
ethereumoptimism/fault-detector latest e9c68fe0e76e 3 weeks ago 1.28GB
ethereumoptimism/deployer latest 45b0cb0b15bf 3 weeks ago 1.28GB
ethereumoptimism/data-transport-layer latest e66ac6868d18 3 weeks ago 1.28GB
ethereumoptimism/replica-healthcheck latest 75d4ec10bce2 4 weeks ago 1.19GB
ethereumoptimism/l2geth latest dfeabbadcd62 4 weeks ago 41.4MB
ethereumoptimism/hardhat latest aeac162bde6b 10 months ago 271MB
ethereumoptimism/integration-tests latest 3ceed3144a81 14 months ago 2.22GB
ops % docker-compose up
Later submit tx:
% curl -X POST --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction", "params":["0xf8688001830186a09423618e81e3f5cdf7f54c3d65f7fbc0abf5b21e8f880de0b6b3a7640000801ba0785e949640b432c31d4f26dc4d3f24875694fdc02715d8ad711d8ade09937c09a02d45d83751e8dc38c90c47a15f1f1b2e26b1b2d698fe3919d84b12be56b08184"], "id": 1001}' -H "Content-type: application/json" http://localhost:8545/ | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 398 100 106 100 292 1828 5037 --:--:-- --:--:-- --:--:-- 7509
{
"jsonrpc": "2.0",
"id": 1001,
"result": "0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b"
}
Logs l2geth, replica:
l2geth_1 | TRACE[06-01|21:49:24.398] Sequencer transaction validation hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
l2geth_1 | INFO [06-01|21:49:24.398] Updating latest timestamp timestamp=2022-06-01T21:49:24+0000 unix=1654120164
l2geth_1 | DEBUG[06-01|21:49:24.401] Applying transaction to tip index=21 hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b origin=sequencer
l2geth_1 | TRACE[06-01|21:49:24.401] Waiting for transaction to be added to chain hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
l2geth_1 | DEBUG[06-01|21:49:24.402] Attempting to commit rollup transaction hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
l2geth_1 | DEBUG[06-01|21:49:24.409] Adding L1 fee l1-fee=6633
l2geth_1 | INFO [06-01|21:49:24.413] New block index=21 l1-timestamp=1654120164 l1-blocknumber=40 tx-hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b queue-orign=sequencer gas=21000 fees=2.1e-14 elapsed=10.926ms
l2geth_1 | TRACE[06-01|21:49:24.416] Waiting for slot to sign and propagate delay=0s
l2geth_1 | DEBUG[06-01|21:49:24.423] Persisted trie from memory database nodes=9 size=1.62KiB time=1.566833ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=-3444.00B
l2geth_1 | DEBUG[06-01|21:49:24.424] Miner got new head height=22 block-hash=0xfa51bb56565fd49318dd5440351c01f592b80fe422132dd2e29d871079963386 tx-hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b tx-hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
l2geth_1 | INFO [06-01|21:49:24.425] Submitted transaction fullhash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b recipient=0x23618e81E3f5cdF7f54C3d65f7FBc0aBf5B21E8f
l2geth_1 | DEBUG[06-01|21:49:24.425] Served eth_sendRawTransaction conn=192.168.160.1:60444 reqid=1001 t=32.084959ms
l2geth_1 | DEBUG[06-01|21:49:24.427] Reinjecting stale transactions count=0
dtl_1 | {"level":30,"time":1654120164430,"method":"GET","url":"/enqueue/latest","elapsed":2,"msg":"Served HTTP Request"}
eth_blockNumber (5) | {"level":30,"time":1654120164436,"method":"GET","url":"/enqueue/latest","elapsed":1,"msg":"Served HTTP Request"}
l1_chain_1 | eth_chainId
eth_chainId (2) | DEBUG[06-01|21:49:24.467] Served eth_blockNumber conn=192.168.160.6:36482 reqid=5931 t=333.625µs
dtl_1 | {"level":30,"time":1654120164470,"fromBlock":21,"toBlock":22,"msg":"Synchronizing unconfirmed transactions from Layer 2 (Optimism)"}
l1_chain_1 | eth_blockNumber
l2geth_1 | DEBUG[06-01|21:49:24.610] Served eth_getBlockRange conn=192.168.160.6:36486 reqid=1 t=4.448834ms
l1_chain_1 | eth_getBlockByNumber
dtl_1 | {"level":30,"time":1654120164847,"method":"GET","url":"/eth/context/latest","elapsed":407,"msg":"Served HTTP Request"}
dtl_1 | {"level":30,"time":1654120164894,"method":"GET","url":"/batch/transaction/latest","elapsed":19,"msg":"Served HTTP Request"}
dtl_1 | {"level":30,"time":1654120164955,"method":"GET","url":"/transaction/latest?backend=l2","elapsed":88,"msg":"Served HTTP Request"}
l1_chain_1 | eth_blockNumber
dtl_1 | {"level":30,"time":1654120165008,"method":"GET","url":"/transaction/latest?backend=l2","elapsed":14,"msg":"Served HTTP Request"}
dtl_1 | {"level":30,"time":1654120165018,"method":"GET","url":"/enqueue/latest","elapsed":42,"msg":"Served HTTP Request"}
eth_blockNumber (2) | {"level":30,"time":1654120165035,"method":"GET","url":"/enqueue/latest","elapsed":6,"msg":"Served HTTP Request"}
eth_chainId (2) | eth_chainId
l1_chain_1 | eth_getBlockByNumber
dtl_1 | {"level":30,"time":1654120165121,"method":"GET","url":"/eth/context/latest","elapsed":81,"msg":"Served HTTP Request"}
l1_chain_1 | eth_blockNumber
l2geth_1 | DEBUG[06-01|21:49:25.161] Served eth_blockNumber conn=192.168.160.6:36502 reqid=5932 t=156.125µs
dtl_1 | {"level":30,"time":1654120165165,"currentL2Block":22,"targetL2Block":22,"msg":"All Layer 2 (Optimism) transactions are synchronized"}
dtl_1 | {"level":30,"time":1654120165205,"method":"GET","url":"/transaction/latest?backend=l2","elapsed":1,"msg":"Served HTTP Request"}
replica_1 | INFO [06-01|21:49:25.208] Syncing transaction range start=21 end=21 backend=l2
dtl_1 | {"level":30,"time":1654120165215,"method":"GET","url":"/transaction/index/21?backend=l2","elapsed":1,"msg":"Served HTTP Request"}
replica_1 | TRACE[06-01|21:49:25.218] Applying indexed transaction index=21
replica_1 | DEBUG[06-01|21:49:25.222] Applying transaction to tip index=21 hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b origin=sequencer
replica_1 | TRACE[06-01|21:49:25.223] Waiting for transaction to be added to chain hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
replica_1 | DEBUG[06-01|21:49:25.223] Attempting to commit rollup transaction hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
dtl_1 | {"level":30,"time":1654120165230,"method":"GET","url":"/batch/transaction/latest","elapsed":1,"msg":"Served HTTP Request"}
replica_1 | DEBUG[06-01|21:49:25.239] Adding L1 fee l1-fee=6633
replica_1 | INFO [06-01|21:49:25.243] New block index=21 l1-timestamp=1654120164 l1-blocknumber=40 tx-hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b queue-orign=sequencer gas=21000 fees=2.1e-14 elapsed=20.052ms
eth_blockNumber (2) | TRACE[06-01|21:49:25.246] Waiting for slot to sign and propagate delay=0s
replica_1 | DEBUG[06-01|21:49:25.252] Persisted trie from memory database nodes=9 size=1.62KiB time=951.125µs gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=-3444.00B
replica_1 | DEBUG[06-01|21:49:25.253] Miner got new head height=22 block-hash=0xfa51bb56565fd49318dd5440351c01f592b80fe422132dd2e29d871079963386 tx-hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b tx-hash=0x05d0258ae858446fce06eec1cc686702bdc18bd3b5f7b0805da3a10f3282ed2b
replica_1 | DEBUG[06-01|21:49:25.254] Reinjecting stale transactions count=0
Logs batch_submitter:
% docker-compose up batch_submitter
ops_l1_chain_1 is up-to-date
ops_deployer_1 is up-to-date
ops_l2geth_1 is up-to-date
Starting ops_batch_submitter_1 ... done
Attaching to ops_batch_submitter_1
batch_submitter_1 | INFO [06-02|00:50:17.181] Config parsed min_tx_size=32 max_tx_size=90000
batch_submitter_1 | INFO [06-02|00:50:17.183] Initializing batch submitter
batch_submitter_1 | INFO [06-02|00:50:17.186] Sequencer wallet params parsed successfully wallet_address=0x70997970C51812dc3A010C7d01b50e0d17dc79C8 contract_address=0xCf7Ed3AccA5a467e9e704C703E8D87F634fB0Fc9
batch_submitter_1 | INFO [06-02|00:50:17.187] Proposer wallet params parsed successfully wallet_address=0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC contract_address=0xDc64a140Aa3E981100a9becA4E685f962f0cF6C9
batch_submitter_1 | INFO [06-02|00:50:17.277] Starting batch submitter
batch_submitter_1 | INFO [06-02|00:50:17.278] Batch submitter started
batch_submitter_1 | INFO [06-02|00:50:17.794] Proposer fetching current block range
batch_submitter_1 | INFO [06-02|00:50:17.796] Sequencer fetching current block range
batch_submitter_1 | INFO [06-02|00:50:17.891] Sequencer block range start=1 end=31
batch_submitter_1 | INFO [06-02|00:50:18.012] Sequencer crafting batch tx start=1 end=31 nonce=1 type=ZLIB
batch_submitter_1 | INFO [06-02|00:50:18.103] Proposer no updates start=1 end=1
batch_submitter_1 | INFO [06-02|00:50:18.218] Sequencer testing batch size calldata_size=971 min_tx_size=32 max_tx_size=90000
batch_submitter_1 | INFO [06-02|00:50:18.218] Sequencer batch constructed num_txs=30 final_size=971 batch_type=ZLIB
batch_submitter_1 | INFO [06-02|00:50:18.755] Proposer fetching current block range
batch_submitter_1 | INFO [06-02|00:50:18.759] Sequencer updating batch tx gas price start=1 end=31 nonce=1
batch_submitter_1 | WARN [06-02|00:50:18.769] Sequencer eth_maxPriorityFeePerGas is unsupported by current backend, using fallback gasTipCap
batch_submitter_1 | panic: runtime error: invalid memory address or nil pointer dereference
batch_submitter_1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x51c873]
batch_submitter_1 |
batch_submitter_1 | goroutine 95 [running]:
batch_submitter_1 | math/big.(*Int).Mul(0xc000149978, 0x0, 0xc000149998)
batch_submitter_1 | /usr/local/go/src/math/big/int.go:168 +0x93
batch_submitter_1 | github.com/ethereum-optimism/optimism/bss-core/txmgr.CalcGasFeeCap(0xc00030c058?, 0xda9278?)
batch_submitter_1 | /go/bss-core/txmgr/txmgr.go:335 +0x7c
batch_submitter_1 | github.com/ethereum-optimism/optimism/batch-submitter/drivers/sequencer.(*Driver).UpdateGasPrice(0xc0004e6b00, {0xda9278?, 0xc00035f3c0}, 0xc00015aa20)
batch_submitter_1 | /go/batch-submitter/drivers/sequencer/driver.go:340 +0x13d
batch_submitter_1 | github.com/ethereum-optimism/optimism/bss-core.(*Service).eventLoop.func1({0xda9278, 0xc00035f3c0})
batch_submitter_1 | /go/bss-core/service.go:209 +0x192
batch_submitter_1 | github.com/ethereum-optimism/optimism/bss-core/txmgr.(*SimpleTxManager).Send.func1()
batch_submitter_1 | /go/bss-core/txmgr/txmgr.go:141 +0xec
batch_submitter_1 | created by github.com/ethereum-optimism/optimism/bss-core/txmgr.(*SimpleTxManager).Send
batch_submitter_1 | /go/bss-core/txmgr/txmgr.go:205 +0x2f5
ops_batch_submitter_1 exited with code 2
% docker exec -it ops_l2geth_1 ./geth version
Geth
Version: 1.9.10-stable
Architecture: amd64
Protocol Versions: [64 63]
Go Version: go1.18
Operating System: linux
GOPATH=
GOROOT=/usr/local/go
@smartcontracts I believe the issue is on the hardhat image, the image which is on dockerhub:
ethereumoptimism/hardhat latest aeac162bde6b 10 months ago 271MB
After cd ops/docker/hardhat/ and docker build . -t ethereumoptimism/hardhat I have:
hardhat % docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ethereumoptimism/hardhat latest 868c5bc32085 13 minutes ago 194MB
And it seems this solves the issue. Would it be possible to update the ci pipeline to update it as well and to update the existing image? Thanks!
@matheusbrat When you are submitting a trx, what exactly is the address you are providing in the params?
@bmalepaty I have pasted the raw tx I was using, so it should be easy to use it, no?
I found that this problem may be caused by the the L1 chain service. It doesn't support London Upgrade. We can change its hardhat.config.js.
@smartcontracts I believe the issue is on the hardhat image, the image which is on dockerhub:
ethereumoptimism/hardhat latest aeac162bde6b 10 months ago 271MBAfter
cd ops/docker/hardhat/anddocker build . -t ethereumoptimism/hardhatI have:hardhat % docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE ethereumoptimism/hardhat latest 868c5bc32085 13 minutes ago 194MBAnd it seems this solves the issue. Would it be possible to update the ci pipeline to update it as well and to update the existing image? Thanks!
I was running into this same issue, and this fixed it for me.
https://github.com/ethereum-optimism/optimism/pull/3339 should fix this