erigon
erigon copied to clipboard
`Waiting for Consensus Layer...` takes forever
System information
Erigon version: ./erigon --version
OS & Version: Linux
Commit hash : 32bd69e5316050005e34448ec6b0165f97173d50
Expected behaviour
Not get stucked over Waiting for Consensus Layer...
.
Actual behaviour
Stuck over Waiting for Consensus Layer...
Steps to reproduce the behaviour
Full archive sync with 32bd69e5316050005e34448ec6b0165f97173d50, no mods.
Backtrace
There was this warning WARN[09-28|14:23:02.861] TxLookup pruning, empty block body height=14999999
, right before starting Waiting for Consensus Layer
. Not sure what that is about.
INFO[09-28|14:11:01.991] [14/16 TxLookup] ETL [1/2] Extracting from=CanonicalHeader block=15422426 alloc=5.0GB sys=8.8GB
INFO[09-28|14:11:12.012] [p2p] GoodPeers eth66=93
INFO[09-28|14:11:12.829] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=4.5GB sys=8.8GB
INFO[09-28|14:11:30.372] Flushed buffer file name=/pool/ethereum/erigon-db/etl-temp/erigon-sortable-buf-2709626317 alloc=4.0GB sys=8.8GB
INFO[09-28|14:11:31.991] [14/16 TxLookup] ETL [1/2] Extracting from=CanonicalHeader block=15440373 alloc=4.2GB sys=8.8GB
INFO[09-28|14:12:06.452] Flushed buffer file name=/pool/ethereum/erigon-db/etl-temp/erigon-sortable-buf-1950804248 alloc=4.3GB sys=8.8GB
INFO[09-28|14:12:06.453] [14/16 TxLookup] ETL [1/2] Extracting from=CanonicalHeader block=15462888 alloc=4.3GB sys=8.8GB
INFO[09-28|14:12:12.836] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=2.9GB sys=8.8GB
INFO[09-28|14:12:31.990] [14/16 TxLookup] ETL [1/2] Extracting from=CanonicalHeader block=15484673 alloc=4.1GB sys=8.8GB
INFO[09-28|14:12:40.799] Flushed buffer file name=/pool/ethereum/erigon-db/etl-temp/erigon-sortable-buf-3089265118 alloc=4.4GB sys=8.8GB
INFO[09-28|14:13:01.990] [14/16 TxLookup] ETL [1/2] Extracting from=CanonicalHeader block=15503011 alloc=3.2GB sys=8.8GB
INFO[09-28|14:13:12.012] [p2p] GoodPeers eth66=94
INFO[09-28|14:13:12.852] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=4.6GB sys=8.8GB
INFO[09-28|14:13:16.257] Flushed buffer file name=/pool/ethereum/erigon-db/etl-temp/erigon-sortable-buf-1559182678 alloc=4.7GB sys=8.8GB
INFO[09-28|14:13:31.991] [14/16 TxLookup] ETL [1/2] Extracting from=CanonicalHeader block=15521633 alloc=4.9GB sys=8.8GB
INFO[09-28|14:13:51.880] Flushed buffer file name=/pool/ethereum/erigon-db/etl-temp/erigon-sortable-buf-3746061139 alloc=4.8GB sys=8.8GB
INFO[09-28|14:14:01.152] Flushed buffer file name=/pool/ethereum/erigon-db/etl-temp/erigon-sortable-buf-1940986141 alloc=3.6GB sys=8.8GB
INFO[09-28|14:14:12.832] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:14:31.194] [14/16 TxLookup] ETL [2/2] Loading into=BlockTransactionLookup current key=7002e38e... alloc=4.0GB sys=8.8GB
INFO[09-28|14:15:01.195] [14/16 TxLookup] ETL [2/2] Loading into=BlockTransactionLookup current key=e34d5516... alloc=4.6GB sys=8.8GB
INFO[09-28|14:15:09.015] [14/16 TxLookup] etl: temp files removed total size=3.3GB
INFO[09-28|14:15:12.012] [p2p] GoodPeers eth66=93
INFO[09-28|14:15:12.862] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=3.5GB sys=8.8GB
INFO[09-28|14:16:12.829] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=4.1GB sys=8.8GB
INFO[09-28|14:17:12.012] [p2p] GoodPeers eth66=93
INFO[09-28|14:17:12.842] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:18:12.829] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=3.0GB sys=8.8GB
INFO[09-28|14:19:12.012] [p2p] GoodPeers eth66=95
INFO[09-28|14:19:12.842] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:20:15.604] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=2.8GB sys=8.8GB
INFO[09-28|14:21:12.012] [p2p] GoodPeers eth66=95
INFO[09-28|14:21:16.188] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=4.6GB sys=8.8GB
INFO[09-28|14:22:12.838] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:23:02.827] [14/16 TxLookup] DONE in=23m30.842619742s
WARN[09-28|14:23:02.861] TxLookup pruning, empty block body height=14999999
INFO[09-28|14:23:02.899] Timings (slower than 50ms) Snapshots=4.497s LogIndex=6h44m14.632s TxLookup=23m30.842s
INFO[09-28|14:23:02.908] RPC Daemon notified of new headers from=15536369 to=15537393 header sending=6.733929ms log sending=3.066µs
INFO[09-28|14:23:02.909] [2/16 Headers] Waiting for Consensus Layer...
INFO[09-28|14:23:12.012] [p2p] GoodPeers eth66=95
INFO[09-28|14:23:12.845] [txpool] stat block=15537393 pending=8978 baseFee=0 queued=30000 alloc=4.1GB sys=8.8GB
INFO[09-28|14:24:12.835] [txpool] stat block=15537393 pending=8977 baseFee=0 queued=30000 alloc=3.6GB sys=8.8GB
INFO[09-28|14:25:12.012] [p2p] GoodPeers eth66=94
INFO[09-28|14:25:14.735] [txpool] stat block=15537393 pending=8977 baseFee=0 queued=30000 alloc=2.6GB sys=8.8GB
INFO[09-28|14:26:12.836] [txpool] stat block=15537393 pending=8977 baseFee=0 queued=30000 alloc=3.3GB sys=8.8GB
INFO[09-28|14:27:12.012] [p2p] GoodPeers eth66=94
INFO[09-28|14:27:12.848] [txpool] stat block=15537393 pending=8970 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:28:12.853] [txpool] stat block=15537393 pending=8970 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:29:12.012] [p2p] GoodPeers eth66=93
INFO[09-28|14:29:12.839] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:30:17.109] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:31:12.012] [p2p] GoodPeers eth66=93
INFO[09-28|14:31:12.838] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:32:12.840] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=3.0GB sys=8.8GB
INFO[09-28|14:33:12.012] [p2p] GoodPeers eth66=96
INFO[09-28|14:33:12.843] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:34:12.837] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:35:12.012] [p2p] GoodPeers eth66=96
INFO[09-28|14:35:12.839] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:36:12.831] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:37:12.012] [p2p] GoodPeers eth66=98
INFO[09-28|14:37:12.844] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:38:12.836] [txpool] stat block=15537393 pending=8969 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:39:12.012] [p2p] GoodPeers eth66=98
INFO[09-28|14:39:12.841] [txpool] stat block=15537393 pending=8967 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:40:12.846] [txpool] stat block=15537393 pending=8967 baseFee=0 queued=30000 alloc=3.0GB sys=8.8GB
INFO[09-28|14:41:12.012] [p2p] GoodPeers eth66=98
INFO[09-28|14:41:12.847] [txpool] stat block=15537393 pending=8967 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:42:12.830] [txpool] stat block=15537393 pending=8967 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:43:12.012] [p2p] GoodPeers eth66=96
INFO[09-28|14:43:12.845] [txpool] stat block=15537393 pending=8967 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:44:12.839] [txpool] stat block=15537393 pending=8965 baseFee=0 queued=30000 alloc=3.1GB sys=8.8GB
INFO[09-28|14:45:12.012] [p2p] GoodPeers eth66=95
INFO[09-28|14:45:13.379] [txpool] stat block=15537393 pending=8965 baseFee=0 queued=30000 alloc=2.5GB sys=8.8GB
INFO[09-28|14:46:12.838] [txpool] stat block=15537393 pending=8964 baseFee=0 queued=30000 alloc=3.3GB sys=8.8GB
INFO[09-28|14:47:12.012] [p2p] GoodPeers eth66=95
INFO[09-28|14:47:12.849] [txpool] stat block=15537393 pending=8963 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:48:12.842] [txpool] stat block=15537393 pending=8963 baseFee=0 queued=30000 alloc=3.0GB sys=8.8GB
INFO[09-28|14:49:12.012] [p2p] GoodPeers eth66=97
INFO[09-28|14:49:12.841] [txpool] stat block=15537393 pending=8963 baseFee=0 queued=30000 alloc=2.4GB sys=8.8GB
INFO[09-28|14:50:12.836] [txpool] stat block=15537393 pending=8963 baseFee=0 queued=30000 alloc=3.0GB sys=8.8GB
...
You need install ConsensusLayer
I woke up this morning to the same scenario. It's not clear to me looking at Erigon's README.md what should be done?
In my case I'm using a stock Docker Compose setup to run Erigon - I thought this approach would give me all the required deps to fire up the node.
After TheMerge - Erigon it's an Execution Layer. Need connect some Consensus Client to Erigon:
https://ethereum.org/en/developers/docs/nodes-and-clients/#consensus-clients
PoW chains still can be synced by Erigon-only.
@AskAlexSharov - Thanks for the follow up!
I'm not sure if I have things setup correctly, but it seems like Prysm is processing blocks...
I was not able to get Lighthouse + Erigon to work together, seems to be authentication related. (details below)
FWIW - Some ideas that might make it easier to get started w. Erigon post-merge:
- Expand on the documentation surrounding CL/EL setup
- Maybe offer a few additional docker-compose examples. It's possible to create secondary compose files that users could opt into as a quickstart & example of how to wire clients up.
Launch Erigon + Lighthouse
(not working - see notes below)
docker compose -f docker-compose.yaml -f docker-compose.lighthouse.yaml up -d
docker-compose.lighthouse.yaml
version: '2.2'
services:
lighthouse:
image: sigp/lighthouse:latest-modern
ports: [ "5052:5052/tcp", "5052:5052/udp", "9000:9000/tcp", "9000:9000/udp" ]
command: |
lighthouse beacon
--http
--execution-endpoint http://erigon:8551
--execution-jwt /home/erigon/.local/share/erigon/jwt.hex
volumes:
- ${XDG_DATA_HOME:-~/.local/share}/lighthouse:/root/.lighthouse:rw
- ${XDG_DATA_HOME:-~/.local/share}/erigon:/home/erigon/.local/share/erigon:ro
Launch Erigon + Prysm
docker compose -f docker-compose.yaml -f docker-compose.prysm.yaml up -d
docker-compose.prysm.yml
version: '2.2'
services:
prysm:
image: gcr.io/prysmaticlabs/prysm/beacon-chain:stable
command: |
--accept-terms-of-use
--datadir=/data
--jwt-secret=/home/erigon/.local/share/erigon/jwt.hex
--rpc-host=0.0.0.0
--grpc-gateway-host=0.0.0.0
--monitoring-host=0.0.0.0
--execution-endpoint=http://erigon:8551
ports:
- "4000:4000"
- "1300:1300"
- "1200:1200/udp"
volumes:
- ${XDG_DATA_HOME:-~/.local/share}/prysm:/data:rw
- ${XDG_DATA_HOME:-~/.local/share}/erigon:/home/erigon/.local/share/erigon:ro
Notes:
- For the examples above, I modified the default Erigon
docker-compose.yaml
. Add the following--authrpc.addr=0.0.0.0
to theerigon
service. - For some reason Lighthouse would refuse to connect to Erigon. This seems to be JWT related, but I could not track down the root cause. It seems like at least some others have hit similar issues in the past.
- Reddit discussion about Erigon + Lighhouse
- https://github.com/sigp/lighthouse/issues/3528
@mjpowersjr were you able to get this working ? i'm stuck at same point !
@mjpowersjr @chetanbothra A common cause for the InvalidToken
errors is that the EL isn't configured to accept requests on a domain other than localhost. In your example setting --authrpc.vhosts erigon
in Erigon's config should fix it (as described in the README: https://github.com/ledgerwatch/erigon#beacon-chain-consensus-layer).
Thank you @michaelsproul , I tried this but still stuck,
These are steps i tried
Docker-compose service for erigon
services:
erigon:
image: thorax/erigon:${TAG:-latest}
build:
args:
UID: ${DOCKER_UID:-1000}
GID: ${DOCKER_GID:-1000}
context: .
command: |
erigon ${ERIGON_FLAGS-} --private.api.addr=0.0.0.0:9090
--sentry.api.addr=sentry:9091 --downloader.api.addr=downloader:9093 --txpool.disable
--metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --pprof --pprof.addr=0.0.0.0 --pprof.port=6061
--authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex --authrpc.vhosts erigon --datadir=/home/erigon/.local/share/erigon
ports: [ "8551:8551" ]
volumes:
# It's ok to mount sub-dirs of "datadir" to different drives
- ${XDG_DATA_HOME:-~/.local/share}/erigon:/home/erigon/.local/share/erigon
restart: unless-stopped
mem_swappiness: 0services:
erigon:
image: thorax/erigon:${TAG:-latest}
build:
args:
UID: ${DOCKER_UID:-1000}
GID: ${DOCKER_GID:-1000}
context: .
command: |
erigon ${ERIGON_FLAGS-} --private.api.addr=0.0.0.0:9090
--sentry.api.addr=sentry:9091 --downloader.api.addr=downloader:9093 --txpool.disable
--metrics --metrics.addr=0.0.0.0 --metrics.port=6060 --pprof --pprof.addr=0.0.0.0 --pprof.port=6061
--authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex --authrpc.vhosts erigon --datadir=/home/erigon/.local/share/erigon
ports: [ "8551:8551" ]
volumes:
# It's ok to mount sub-dirs of "datadir" to different drives
- ${XDG_DATA_HOME:-~/.local/share}/erigon:/home/erigon/.local/share/erigon
restart: unless-stopped
mem_swappiness: 0
and then after starting docker , here are my docker logs for erigon
docker-compose logs -f erigon
erigon-erigon-1 | [INFO] [10-11|04:12:14.638] Starting metrics server addr=http://0.0.0.0:6060/debug/metrics/prometheus
erigon-erigon-1 | [INFO] [10-11|04:12:14.638] Starting pprof server cpu="go tool pprof -lines -http=: http://0.0.0.0:6061/debug/pprof/profile?seconds=20" heap="go tool pprof -lines -http=: http://0.0.0.0:6061/debug/pprof/heap"
erigon-erigon-1 | [INFO] [10-11|04:12:14.638] Build info git_branch=HEAD git_tag=v2.27.0-dirty git_commit=6d8b304eae78cef7aa3f3eb7ed584559bdfe7dbb
erigon-erigon-1 | [INFO] [10-11|04:12:14.638] Starting Erigon on Ethereum mainnet...
erigon-erigon-1 | [INFO] [10-11|04:12:14.638] Maximum peer count ETH=100 total=100
erigon-erigon-1 | [INFO] [10-11|04:12:14.639] starting HTTP APIs APIs=eth,erigon,engine
erigon-erigon-1 | [INFO] [10-11|04:12:14.639] Set global gas cap cap=50000000
erigon-erigon-1 | [INFO] [10-11|04:12:14.671] Opening Database label=chaindata path=/home/erigon/.local/share/erigon/chaindata
erigon-erigon-1 | [INFO] [10-11|04:12:14.674] Initialised chain configuration config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, DAO Support: true, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
erigon-erigon-1 | [INFO] [10-11|04:12:14.674] Effective prune_flags= snapshot_flags="--snapshots=true" history.v3=false
erigon-erigon-1 | [INFO] [10-11|04:12:14.674] Initialising Ethereum protocol network=1
erigon-erigon-1 | [INFO] [10-11|04:12:14.674] Disk storage enabled for ethash DAGs dir=/home/erigon/.local/share/erigon/ethash-dags count=2
erigon-erigon-1 | [INFO] [10-11|04:12:14.674] Starting private RPC server on=0.0.0.0:9090
erigon-erigon-1 | [INFO] [10-11|04:12:14.675] new subscription to logs established
erigon-erigon-1 | [INFO] [10-11|04:12:14.675] rpc filters: subscribing to Erigon events
erigon-erigon-1 | [INFO] [10-11|04:12:14.675] new subscription to newHeaders established
erigon-erigon-1 | [INFO] [10-11|04:12:14.675] Reading JWT secret path=/home/erigon/.local/share/erigon/jwt.hex
erigon-erigon-1 | [INFO] [10-11|04:12:14.675] HTTP endpoint opened for Engine API url=localhost:8551 ws=true ws.compression=true
erigon-erigon-1 | [INFO] [10-11|04:12:14.676] HTTP endpoint opened url=localhost:8545 ws=false ws.compression=true grpc=false
erigon-erigon-1 | [INFO] [10-11|04:12:14.687] [1/16 Snapshots] Fetching torrent files metadata
erigon-erigon-1 | [INFO] [10-11|04:12:17.820] new subscription to newHeaders established
erigon-erigon-1 | [INFO] [10-11|04:12:19.466] [Snapshots] Blocks Stat blocks=15000k indices=15000k alloc=2.2GB sys=2.4GB
erigon-erigon-1 | [INFO] [10-11|04:12:19.922] [2/16 Headers] Waiting for Consensus Layer...
stays here forever !
@chetanbothra are you running a consensus client too? you need a consensus client like Nimbus, Teku, Lodestar, Prysm or Lighthouse.
as per readme Erigon can be used as an Execution Layer (EL) for Consensus Layer clients (CL). Default configuration is OK. so i didnt add anything as such .
Since I'm new and learning, Do u have a reference on how to add consensus client to existing erigon
@chetanbothra
A lot of the guides are currently geared towards staking, but if you ignore the parts about generating keys and running a validator client they could be good:
- Lighthouse guide: https://someresat.medium.com/guide-to-staking-on-ethereum-ubuntu-lighthouse-773f5d982e03
The clients also have docs:
- Prysm: https://docs.prylabs.network/docs/install/install-with-script
- Teku: https://docs.teku.consensys.net/en/latest/HowTo/Get-Started/Run-Teku/
- Lighthouse: https://lighthouse-book.sigmaprime.io/merge-migration.html
- Geth's docs about consensus clients: https://geth.ethereum.org/docs/interface/consensus-clients
Eth-Docker is another option if you want a docker-based set up:
- https://eth-docker.net/docs/About/Overview/
HI @mjpowersjr Were you able to fix this? Erigon + Lighthouse in docker compose.
Having same issue with Prysm. Any updates on this?
I'm encountering this issue during the initial sync, which failed halfway through snapshot 7/16 (out of ram). But upon restarting sync (with more ram/swap space), it skips the remaining snapshots and is stuck at the waiting for consensus layer
step. Afaik, erigon client should be able to fully sync first?
@0xSage I m not a expert but i would advice two additional flags if not added already in erigon
- --batchSize=512M ( to avoid OOM)
- --download.verify
This issue is stale because it has been open for 40 days with no activity. Remove stale label or comment, or this will be closed in 7 days.
This issue was closed because it has been stalled for 7 days with no activity.
HI @mjpowersjr Were you able to fix this? Erigon + Lighthouse in docker compose.
Sorry for the late reply. Unfortunately no - I was not able to resolve the issue. I left my server with spin with a high load average for at least a week hoping things would work out. Eventually I shut everything down until I have spare cycles to revisit.
FWIW I don't know if my hardware is under-powered or I'm just not configuring something correctly. I tried a few different consensus layer clients w. different errors.
When I resume, I'm going to take a close look @ the RocketPool tutorials and see if they have any good tips that might help. (They don't seem to have instructions for erigon, but maybe the consensus layer sections will help).
https://docs.rocketpool.net/guides/node/eth-clients.html#nethermind
Good Luck and sorry I couldn't help more!
@mjpowersjr lifesaver.
I've found the documentation lacking and confusing.
Just run the erigon node with --internalcl
flag; no need for an external consensus layer.
--internalcl
Just run the erigon node with
--internalcl
flag; no need for an external consensus layer.--internalcl
Thanks @tanuj-deqode ! - This feature is an exciting addition to Erigon, I suspect this will be a good way to go for many. (This wasn't available when most of the original discussions above took place)
Unfortunately I'm seeing the same Waiting for Consensus Layer...
while using the --internalcl
flag...
Experiencing the same with --internalcl
flag set.
Version? Network?
Goerli, erigon version 2.43.0-stable-1106475e
I enabled lighthouse after removing the internalcl
flag and everything has moved past block 8499999
I wonder if it's an issue using the internal CL when sync'ing from scratch or something?