erigon
erigon copied to clipboard
Published
20 hours ago •
erigontech
erigontech
[Erigon + Prysm] Infinite Loop After Runtime Error `runtime error: invalid memory address or nil pointer dereference`
trafficstars
System information
Erigon version: 2022.09.1-alpha-4067b7c4
Prysm version: 3.1.1
OS & Version: Linux ethereum-erigon 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Expected behaviour
Erigon Syncing with Mainnet post-merge.
Actual behaviour
Erigon keeps trying to catch-up but always fails due to unwinding after this log message:
WARN[09-15|20:50:04.290] Unexpected header hash=0x78c1c271074042eaeb85d9650fd4c486cd58ee464810ed1719623620e858cd79 expected=0x74e11e00f1e45812dbf9da130d819acc5174926c494963ef1e6b5fe8709b25c5
Erigon keeps working for ~1 hour then this error log show up:
EROR[09-15|22:07:03.399] Staged Sync err="runtime error: invalid memory address or nil pointer dereference, trace: [stageloop.go:129 panic.go:890 panic.go:260 signal_unix.go:835 fork_validator.go:279 fork_validator.go:216 stage_headers.go:673 collector.go:276 collector.go:144 stage_headers.go:686 stage_headers.go:732 stage_headers.go:195 stage_headers.go:163 default_stages.go:20 sync.go:352 sync.go:255 stageloop.go:163 stageloop.go:84 asm_amd64.s:1594]"
Few seconds later, Erigon falls into infinite loop with this logs:
INFO[09-15|22:08:02.973] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=65.39µs log sending=434ns
INFO[09-15|22:08:02.973] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:08:02.973] [1/15 Headers] Fork choice missing header with hash 77052c9c241525ec9458932fede7bd346dcd5f0ac8708b8e493ea48566782128
INFO[09-15|22:08:02.975] Commit cycle in=16.906µs
YAML Config
# -----------------
# Basics
# -----------------
chain: "mainnet"
datadir: "/mnt/data"
history.v2: "false"
watch-the-burn: "true"
snapshots: "true"
healthcheck: "true"
# -----------------
# P2P
# -----------------
nat: "extip:<SNIP>"
port: "<SNIP>"
maxpeers: "200"
v5disc: "true"
# -----------------
# Json-RPC Over HTTP
# -----------------
http: "true"
http.trace: "false"
http.compression: "false"
http.addr: "0.0.0.0"
http.port: "8545"
http.api: "eth,erigon,engine,web3,net,debug,trace,txpool,db,parity"
http.vhosts: "*"
http.corsdomain: "*"
# -----------------
# Json-RPC Over WebSocket
# -----------------
ws: "true"
ws.compression: "false"
# -----------------
# Engine API
# -----------------
authrpc.addr: "0.0.0.0"
authrpc.port: "8551"
authrpc.vhosts: "*"
# -----------------
# Private API
# -----------------
private.api.addr: "0.0.0.0:9090"
# -----------------
# Memory Pool
# -----------------
txpool.globalslots: "1000000"
txpool.globalbasefeeslots: "1000000"
txpool.globalqueue: "1000000"
txpool.lifetime: "720h"
# -----------------
# Downloader
# -----------------
torrent.maxpeers: "1"
torrent.upload.rate: "1kb"
torrent.download.rate: "1mb"
no-downloader: "false"
# -----------------
# Metrics
# -----------------
metrics.addr: "0.0.0.0"
metrics.port: "6070"
metrics: "true"
metrics.expensive: "true"
# -----------------
# pprof
# -----------------
pprof.addr: "0.0.0.0"
pprof.port: "6060"
pprof: "true"
Logs
/home/texnomic/erigon/build/bin/erigon --config /mnt/data/Erigon.yaml
INFO[09-15|22:04:36.140] Commit cycle in=4.40243ms
INFO[09-15|22:04:36.140] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=49.86µs log sending=429ns
INFO[09-15|22:04:36.140] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:04:36.141] [1/15 Headers] Fork choice missing header with hash 487e9eb99e9494c1a7b59c5dcc03dd93968ec3e98fd44753941725d37f4940f3
INFO[09-15|22:04:36.141] [1/15 Headers] Downloading PoS headers... height=0 hash=0x487e9eb99e9494c1a7b59c5dcc03dd93968ec3e98fd44753941725d37f4940f3 requestId=439
INFO[09-15|22:04:36.142] Commit cycle in=9.055µs
INFO[09-15|22:04:36.142] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=33.236µs log sending=196ns
INFO[09-15|22:04:36.142] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:04:48.141] [4/7 Senders] Started from=15541313 to=15541338
INFO[09-15|22:04:48.415] [5/7 Execution] Blocks execution from=15541313 to=15541338
INFO[09-15|22:04:51.102] [5/7 Execution] Completed on block=15541338
INFO[09-15|22:04:51.102] [6/7 HashState] Promoting plain state from=15541313 to=15541338
INFO[09-15|22:04:51.103] [6/7 HashState] Incremental promotion from=15541313 to=15541338 codes=true csbucket=AccountChangeSet
INFO[09-15|22:04:51.219] [6/7 HashState] Incremental promotion from=15541313 to=15541338 codes=false csbucket=AccountChangeSet
INFO[09-15|22:04:51.297] [6/7 HashState] Incremental promotion from=15541313 to=15541338 codes=false csbucket=StorageChangeSet
INFO[09-15|22:04:51.452] [7/7 IntermediateHashes] Generating intermediate hashes from=15541313 to=15541338
INFO[09-15|22:04:55.738] Timings (slower than 50ms) Bodies=1.337s Senders=273ms Execution=2.687s HashState=349ms IntermediateHashes=4.285s
INFO[09-15|22:04:55.738] Tables PlainState=904.0KB AccountChangeSet=492.0KB StorageChangeSet=748.0KB BlockTransaction=2.1MB TransactionLog=1.7MB FreeList=0B
INFO[09-15|22:05:00.015] [4/7 Senders] Started from=15541313 to=15541339
INFO[09-15|22:05:00.278] [5/7 Execution] Blocks execution from=15541313 to=15541339
INFO[09-15|22:05:03.089] [5/7 Execution] Completed on block=15541339
INFO[09-15|22:05:03.089] [6/7 HashState] Promoting plain state from=15541313 to=15541339
INFO[09-15|22:05:03.089] [6/7 HashState] Incremental promotion from=15541313 to=15541339 codes=true csbucket=AccountChangeSet
INFO[09-15|22:05:03.244] [6/7 HashState] Incremental promotion from=15541313 to=15541339 codes=false csbucket=AccountChangeSet
INFO[09-15|22:05:03.337] [6/7 HashState] Incremental promotion from=15541313 to=15541339 codes=false csbucket=StorageChangeSet
INFO[09-15|22:05:03.517] [7/7 IntermediateHashes] Generating intermediate hashes from=15541313 to=15541339
INFO[09-15|22:05:06.738] [p2p] GoodPeers eth66=100
INFO[09-15|22:05:07.976] [txpool] stat block=15541337 pending=4416 baseFee=150310 queued=73876 alloc=4.2GB sys=6.7GB
INFO[09-15|22:05:08.136] Timings (slower than 50ms) Bodies=4.255s Senders=262ms Execution=2.81s HashState=428ms IntermediateHashes=4.618s
INFO[09-15|22:05:08.136] Tables PlainState=944.0KB AccountChangeSet=512.0KB StorageChangeSet=776.0KB BlockTransaction=2.2MB TransactionLog=1.7MB FreeList=0B
INFO[09-15|22:05:12.279] [4/7 Senders] Started from=15541313 to=15541340
INFO[09-15|22:05:12.511] [5/7 Execution] Blocks execution from=15541313 to=15541340
INFO[09-15|22:05:15.305] [5/7 Execution] Completed on block=15541340
INFO[09-15|22:05:15.305] [6/7 HashState] Promoting plain state from=15541313 to=15541340
INFO[09-15|22:05:15.305] [6/7 HashState] Incremental promotion from=15541313 to=15541340 codes=true csbucket=AccountChangeSet
INFO[09-15|22:05:15.378] [6/7 HashState] Incremental promotion from=15541313 to=15541340 codes=false csbucket=AccountChangeSet
INFO[09-15|22:05:15.462] [6/7 HashState] Incremental promotion from=15541313 to=15541340 codes=false csbucket=StorageChangeSet
INFO[09-15|22:05:15.616] [7/7 IntermediateHashes] Generating intermediate hashes from=15541313 to=15541340
INFO[09-15|22:05:18.914] Timings (slower than 50ms) Bodies=4.115s Senders=231ms Execution=2.794s HashState=310ms IntermediateHashes=3.298s
INFO[09-15|22:05:18.914] Tables PlainState=960.0KB AccountChangeSet=524.0KB StorageChangeSet=792.0KB BlockTransaction=2.2MB TransactionLog=1.8MB FreeList=0B
INFO[09-15|22:05:38.938] [2/7 Bodies] Wrote block bodies block_num=15541340 delivery/sec=0B wasted/sec=1.8KB alloc=4.8GB sys=6.7GB
INFO[09-15|22:06:07.874] [txpool] stat block=15541337 pending=4938 baseFee=150366 queued=74067 alloc=5.0GB sys=6.7GB
INFO[09-15|22:06:58.937] [2/7 Bodies] DONE in=1m40.00074475s
INFO[09-15|22:06:58.938] [4/7 Senders] Started from=15541313 to=15541340
INFO[09-15|22:06:59.231] [5/7 Execution] Blocks execution from=15541313 to=15541340
INFO[09-15|22:07:01.398] [5/7 Execution] Completed on block=15541340
INFO[09-15|22:07:01.398] [6/7 HashState] Promoting plain state from=15541313 to=15541340
INFO[09-15|22:07:01.398] [6/7 HashState] Incremental promotion from=15541313 to=15541340 codes=true csbucket=AccountChangeSet
INFO[09-15|22:07:01.460] [6/7 HashState] Incremental promotion from=15541313 to=15541340 codes=false csbucket=AccountChangeSet
INFO[09-15|22:07:01.545] [6/7 HashState] Incremental promotion from=15541313 to=15541340 codes=false csbucket=StorageChangeSet
INFO[09-15|22:07:01.701] [7/7 IntermediateHashes] Generating intermediate hashes from=15541313 to=15541340
INFO[09-15|22:07:03.389] Timings (slower than 50ms) Bodies=1m40s Senders=292ms Execution=2.166s HashState=303ms IntermediateHashes=1.687s
INFO[09-15|22:07:03.389] Tables PlainState=960.0KB AccountChangeSet=524.0KB StorageChangeSet=792.0KB BlockTransaction=2.2MB TransactionLog=1.8MB FreeList=0B
EROR[09-15|22:07:03.399] Staged Sync err="runtime error: invalid memory address or nil pointer dereference, trace: [stageloop.go:129 panic.go:890 panic.go:260 signal_unix.go:835 fork_validator.go:279 fork_validator.go:216 stage_headers.go:673 collector.go:276 collector.go:144 stage_headers.go:686 stage_headers.go:732 stage_headers.go:195 stage_headers.go:163 default_stages.go:20 sync.go:352 sync.go:255 stageloop.go:163 stageloop.go:84 asm_amd64.s:1594]"
INFO[09-15|22:07:05.225] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.253] [1/15 Headers] Fork choice missing header with hash cdc8ca6bce99a035daccb663db6364ffc3b990b48eda54618121ec035e559fba
INFO[09-15|22:07:05.254] Commit cycle in=12.904µs
INFO[09-15|22:07:05.254] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=46.194µs log sending=583ns
INFO[09-15|22:07:05.254] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.270] [1/15 Headers] Fork choice missing header with hash dd0416b2201901d8fbfc3d6957dbaa254bda0861d91c87c17acc47951ac1e658
INFO[09-15|22:07:05.270] Commit cycle in=9.012µs
INFO[09-15|22:07:05.270] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=31.846µs log sending=251ns
INFO[09-15|22:07:05.271] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.287] [1/15 Headers] Fork choice missing header with hash bbc0debb08fae0144e182b594e709d033e19ed586d2f55df18802b052e502e09
INFO[09-15|22:07:05.288] Commit cycle in=10.889µs
INFO[09-15|22:07:05.288] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=46.286µs log sending=285ns
INFO[09-15|22:07:05.288] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.304] [1/15 Headers] Fork choice missing header with hash 98676a44e98957c5073e561b3e7db8b4b2a3da72f8b87f319155c50cbd36e7a3
INFO[09-15|22:07:05.304] Commit cycle in=11.676µs
INFO[09-15|22:07:05.305] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=44.021µs log sending=190ns
INFO[09-15|22:07:05.305] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.322] [1/15 Headers] Fork choice missing header with hash bc3a2edeaf78efa9bba71ba3653630fed5f562cdb9d4a372ced1cc940834bdfb
INFO[09-15|22:07:05.323] Commit cycle in=9.792µs
INFO[09-15|22:07:05.323] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=76.134µs log sending=346ns
INFO[09-15|22:07:05.324] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.339] [1/15 Headers] Fork choice missing header with hash 9bbdeab47ac9d4d2a83046a69a8e74cdc7115d5bdb5bb9280c5afefa9549a0d7
INFO[09-15|22:07:05.340] Commit cycle in=9.626µs
INFO[09-15|22:07:05.340] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=43.853µs log sending=266ns
INFO[09-15|22:07:05.340] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.357] [1/15 Headers] Fork choice missing header with hash dc4828bbb7919c969a9f224fcc2291c0471eaf00e2d503fc49cc0c1a288df936
INFO[09-15|22:07:05.357] Commit cycle in=9.57µs
INFO[09-15|22:07:05.357] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=43.022µs log sending=204ns
INFO[09-15|22:07:05.357] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.373] [1/15 Headers] Fork choice missing header with hash cb9bf5796c2f6e6ad6df1670d98af75fc3fda9d31ef49f5190833d450fc67d59
INFO[09-15|22:07:05.374] Commit cycle in=9.364µs
INFO[09-15|22:07:05.374] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=47.283µs log sending=227ns
INFO[09-15|22:07:05.374] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.389] [1/15 Headers] Fork choice missing header with hash 3663425b55681100a9f6299b9e88defd31ef22679a293805080cd52650c09951
INFO[09-15|22:07:05.389] Commit cycle in=12.346µs
INFO[09-15|22:07:05.389] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=44.913µs log sending=197ns
INFO[09-15|22:07:05.389] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.404] [1/15 Headers] Fork choice missing header with hash f0060049a63abbd98ff373ac4204d4cc2e2e37058acdfd00227f23cc126ecc95
INFO[09-15|22:07:05.405] Commit cycle in=9.302µs
INFO[09-15|22:07:05.405] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=34.701µs log sending=219ns
INFO[09-15|22:07:05.405] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.422] [1/15 Headers] Fork choice missing header with hash 1172c0f9d56d0a036ab5d951e42533acef1a21807e367068755f6f227f7b5e33
INFO[09-15|22:07:05.423] Commit cycle in=9.126µs
INFO[09-15|22:07:05.423] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=44.839µs log sending=215ns
INFO[09-15|22:07:05.423] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.439] [1/15 Headers] Fork choice missing header with hash a72459188936b19254ce469426f2187e32ca2d5e1511d2f0068cd5ac1f61e46f
INFO[09-15|22:07:05.440] Commit cycle in=9.343µs
INFO[09-15|22:07:05.440] RPC Daemon notified of new headers from=15541313 to=15541337 header sending=44.936µs log sending=231ns
INFO[09-15|22:07:05.440] [1/15 Headers] Waiting for Beacon Chain...
INFO[09-15|22:07:05.440] [1/15 Headers] Fork choice missing header with hash a72459188936b19254ce469426f2187e32ca2d5e1511d2f0068cd5ac1f61e46f
<Endless Loop>
We also encounter similar runtime error on Staged Sync:
Staged Sync err="runtime error: invalid memory address or nil pointer dereference, trace: [stageloop.go:129 panic.go:844 panic.go:220 signal_unix.go:818 fork_validator.go:279 fork_validator.go:216 stage_headers.go:673 collector.go:276 collector.go:144 stage_headers.go:686 stage_headers.go:732 stage_headers.go:195 stage_headers.go:163 default_stages.go:20 sync.go:352 sync.go:255 stageloop.go:163 stageloop.go:84 asm_amd64.s:1571]"
Is it related to this pending=10000 queued=10000 just before the staged sync error?
Sep 16 01:41:26 erigon-1 erigon[526931]: txpool_1 | [INFO] [09-16|01:41:26.114] [txpool] stat block=15542990 pending=10000 baseFee=10000 queued=10000 cache_keys=2898 alloc=93.2MB sys=735.1MB
We run the same erigon 2022.09.1-alpha-4067b7c4 but with lighthouse v3.1.0-modern.
Here are our logs:
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.653] Timings (slower than 50ms) Execution=259ms IntermediateHashes=638ms
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.653] Tables PlainState=52.0KB AccountChangeSet=24.0KB StorageChangeSet=32.0KB BlockTransaction=72.0KB TransactionLog=68.0KB FreeList=0B
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.655] Timings (slower than 50ms) Headers=11.466s
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.655] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.658] Commit cycle in=3.417301ms
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.659] RPC Daemon notified of new headers from=15542986 to=15542987 header sending=9.787µs log sending=357ns
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.659] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:00 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:00.756] Flushing in-memory state
Sep 16 01:39:01 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:01.518] Timings (slower than 50ms) Headers=202ms CallTraces=191ms AccountHistoryIndex=119ms StorageHistoryIndex=135ms LogIndex=122ms TxLookup=86ms
Sep 16 01:39:01 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:01.518] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:01 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:39:01.598] no pending block found returning latest executed block
Sep 16 01:39:04 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:04.808] Commit cycle in=3.290421732s
Sep 16 01:39:04 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:04.809] RPC Daemon notified of new headers from=15542986 to=15542987 header sending=18.326µs log sending=398ns
Sep 16 01:39:04 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:04.809] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:08 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:39:08.075] [Snapshots] Seeding up=0B/s peers=2 connections=2 files=90
Sep 16 01:39:11 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:39:11.598] no pending block found returning latest executed block
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.175] [5/7 Execution] Completed on block=15542988
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.773] Timings (slower than 50ms) Execution=264ms IntermediateHashes=569ms
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.773] Tables PlainState=48.0KB AccountChangeSet=20.0KB StorageChangeSet=32.0KB BlockTransaction=152.0KB TransactionLog=96.0KB FreeList=0B
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.775] Timings (slower than 50ms) Headers=7.964s
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.775] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.779] Commit cycle in=4.067206ms
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.779] RPC Daemon notified of new headers from=15542987 to=15542988 header sending=11.085µs log sending=284ns
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.779] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:12 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:12.885] Flushing in-memory state
Sep 16 01:39:13 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:13.926] Timings (slower than 50ms) Headers=594ms CallTraces=129ms AccountHistoryIndex=86ms StorageHistoryIndex=173ms LogIndex=107ms TxLookup=54ms
Sep 16 01:39:13 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:13.926] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:16 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:16.695] Commit cycle in=2.768260906s
Sep 16 01:39:16 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:16.695] RPC Daemon notified of new headers from=15542987 to=15542988 header sending=20.712µs log sending=452ns
Sep 16 01:39:16 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:16.695] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:21 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:39:21.597] no pending block found returning latest executed block
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.710] [5/7 Execution] Completed on block=15542989
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.719] Timings (slower than 50ms) Headers=7.022s
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.719] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.723] Commit cycle in=4.097383ms
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.723] RPC Daemon notified of new headers from=15542988 to=15542989 header sending=8.438µs log sending=352ns
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.723] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.811] Flushing in-memory state
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.828] Timings (slower than 50ms) Headers=95ms
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.828] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.868] Commit cycle in=39.503092ms
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.868] RPC Daemon notified of new headers from=15542988 to=15542989 header sending=30.447µs log sending=740ns
Sep 16 01:39:23 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:23.868] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:26 erigon-1 erigon[526931]: txpool_1 | [INFO] [09-16|01:39:26.133] [txpool] stat block=15542989 pending=9010 baseFee=10000 queued=10000 cache_keys=35 alloc=69.9MB sys=735.1MB
Sep 16 01:39:28 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:39:28.075] [Snapshots] Seeding up=0B/s peers=5 connections=5 files=90
Sep 16 01:39:31 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:39:31.597] no pending block found returning latest executed block
Sep 16 01:39:34 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:34.546] Timings (slower than 50ms) Headers=10.677s
Sep 16 01:39:34 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:34.546] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:34 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:34.549] Commit cycle in=3.460015ms
Sep 16 01:39:34 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:34.550] RPC Daemon notified of new headers from=15542989 to=15542990 header sending=12.118µs log sending=389ns
Sep 16 01:39:34 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:34.550] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:41 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:41.085] [5/7 Execution] Completed on block=15542990
Sep 16 01:39:41 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:39:41.598] no pending block found returning latest executed block
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.220] Timings (slower than 50ms) Execution=476ms IntermediateHashes=1.097s
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.220] Tables PlainState=112.0KB AccountChangeSet=32.0KB StorageChangeSet=80.0KB BlockTransaction=116.0KB TransactionLog=136.0KB FreeList=0B
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.221] Timings (slower than 50ms) Headers=7.67s
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.221] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.225] Commit cycle in=3.819186ms
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.225] RPC Daemon notified of new headers from=15542989 to=15542990 header sending=10.778µs log sending=277ns
Sep 16 01:39:42 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:42.226] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:43 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:43.549] Flushing in-memory state
Sep 16 01:39:45 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:45.092] Timings (slower than 50ms) Headers=1.785s CallTraces=275ms AccountHistoryIndex=201ms StorageHistoryIndex=309ms LogIndex=177ms TxLookup=117ms
Sep 16 01:39:45 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:45.092] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:39:48 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:39:48.075] [Snapshots] Seeding up=0B/s peers=2 connections=2 files=90
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.839] Commit cycle in=5.7467234s
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.839] RPC Daemon notified of new headers from=15542989 to=15542990 header sending=10.112µs log sending=376ns
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.840] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.883] [1/15 Headers] Fork choice missing header with hash 8e4c71271465789fb233d362d03b2190e6199a3a7a7bf874b71070639bcd824c
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.883] [1/15 Headers] Downloading PoS headers... height=0 hash=0x8e4c71271465789fb233d362d03b2190e6199a3a7a7bf874b71070639bcd824c requestId=1097
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.883] Commit cycle in=8.213µs
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.884] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=5.408µs log sending=262ns
Sep 16 01:39:50 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:39:50.884] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:39:51 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:39:51.597] no pending block found returning latest executed block
Sep 16 01:39:56 erigon-1 erigon[526931]: txpool_1 | [INFO] [09-16|01:39:56.220] [txpool] stat block=15542990 pending=9092 baseFee=10000 queued=10000 cache_keys=2898 alloc=59.6MB sys=735.1MB
Sep 16 01:40:01 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:40:01.597] no pending block found returning latest executed block
Sep 16 01:40:08 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:40:08.074] [Snapshots] Seeding up=0B/s peers=5 connections=6 files=90
Sep 16 01:40:11 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:40:11.598] no pending block found returning latest executed block
Sep 16 01:40:21 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:40:21.597] no pending block found returning latest executed block
Sep 16 01:40:26 erigon-1 erigon[526931]: txpool_1 | [INFO] [09-16|01:40:26.211] [txpool] stat block=15542990 pending=9431 baseFee=10000 queued=10000 cache_keys=2898 alloc=87.7MB sys=735.1MB
Sep 16 01:40:28 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:40:28.075] [Snapshots] Seeding up=0B/s peers=4 connections=5 files=90
Sep 16 01:40:31 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:40:31.597] no pending block found returning latest executed block
Sep 16 01:40:41 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:40:41.598] no pending block found returning latest executed block
Sep 16 01:40:48 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:40:48.075] [Snapshots] Seeding up=0B/s peers=5 connections=5 files=90
Sep 16 01:40:51 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:40:51.598] no pending block found returning latest executed block
Sep 16 01:40:55 erigon-1 erigon[526931]: txpool_1 | [INFO] [09-16|01:40:55.810] [txpool] stat block=15542990 pending=9809 baseFee=10000 queued=10000 cache_keys=2898 alloc=67.0MB sys=735.1MB
Sep 16 01:41:01 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:41:01.598] no pending block found returning latest executed block
Sep 16 01:41:08 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:41:08.075] [Snapshots] Seeding up=0B/s peers=2 connections=2 files=90
Sep 16 01:41:11 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:41:11.597] no pending block found returning latest executed block
Sep 16 01:41:21 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:41:21.597] no pending block found returning latest executed block
Sep 16 01:41:26 erigon-1 erigon[526931]: txpool_1 | [INFO] [09-16|01:41:26.114] [txpool] stat block=15542990 pending=10000 baseFee=10000 queued=10000 cache_keys=2898 alloc=93.2MB sys=735.1MB
Sep 16 01:41:28 erigon-1 erigon[526931]: downloader_1 | [INFO] [09-16|01:41:28.075] [Snapshots] Seeding up=0B/s peers=4 connections=4 files=90
Sep 16 01:41:31 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:41:31.597] no pending block found returning latest executed block
Sep 16 01:41:41 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:41.485] [2/7 Bodies] DONE in=1m40.000341578s
Sep 16 01:41:41 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:41.486] Timings (slower than 50ms) Bodies=1m40s
Sep 16 01:41:41 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:41.486] Tables PlainState=0B AccountChangeSet=0B StorageChangeSet=0B BlockTransaction=0B TransactionLog=0B FreeList=0B
Sep 16 01:41:41 erigon-1 erigon[526931]: erigon_1 | [EROR] [09-16|01:41:41.494] Staged Sync err="runtime error: invalid memory address or nil pointer dereference, trace: [stageloop.go:129 panic.go:844 panic.go:220 signal_unix.go:818 fork_validator.go:279 fork_validator.go:216 stage_headers.go:673 collector.go:276 collector.go:144 stage_headers.go:686 stage_headers.go:732 stage_headers.go:195 stage_headers.go:163 default_stages.go:20 sync.go:352 sync.go:255 stageloop.go:163 stageloop.go:84 asm_amd64.s:1571]"
Sep 16 01:41:41 erigon-1 erigon[526931]: rpcdaemon_1 | [WARN] [09-16|01:41:41.598] no pending block found returning latest executed block
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.697] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.742] [1/15 Headers] Fork choice missing header with hash 90f6641983a55b7989dc1cc32da2bac2ecbb20f836a0c644b71335bd785861a2
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.743] Commit cycle in=9.139µs
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.743] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=16.888µs log sending=433ns
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.743] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.789] [1/15 Headers] Fork choice missing header with hash a9be64115c94817c250a4066ca7cb79f09596564641499c5a228fdc451cc9177
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.790] Commit cycle in=8.617µs
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.790] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=12.493µs log sending=439ns
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.790] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.840] [1/15 Headers] Fork choice missing header with hash cc96da3390fe323a4df1ef0956a4962cb497c99a807a4e4a72b0498791a1a7dd
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.841] Commit cycle in=7.581µs
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.841] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=8.033µs log sending=508ns
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.841] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.882] [1/15 Headers] Fork choice missing header with hash cf15bc9ab22b07680b2bca6b8820628913af04c505511bf33d7ae0bf9b2e1546
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.882] Commit cycle in=9.121µs
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.883] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=14.844µs log sending=383ns
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.883] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.928] [1/15 Headers] Fork choice missing header with hash a366965ab2381f7eecc84b8208092cc26f29a3133a05a2bae18b56db1f0ca07c
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.929] Commit cycle in=10.821µs
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.929] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=12.139µs log sending=664ns
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.930] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.980] [1/15 Headers] Fork choice missing header with hash 1b91a6fff59422a6909ae4fc982a7c640ca58b5ebd5b06803dbc40a42f24cb9d
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.981] Timings (slower than 50ms) Headers=51ms
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.981] Tables PlainState=73.8GB AccountChangeSet=162.8GB StorageChangeSet=240.9GB BlockTransaction=7.8GB TransactionLog=447.4GB FreeList=132.5MB ReclaimableSpace=132.5GB
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.981] Commit cycle in=11.424µs
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.982] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=24.411µs log sending=631ns
Sep 16 01:41:47 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:47.982] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:48 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:48.023] [1/15 Headers] Fork choice missing header with hash 5b736cdea70a71f184328e2aab7914eb2233544828a113b9e773bd37477fbe27
Sep 16 01:41:48 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:48.024] Commit cycle in=9.907µs
Sep 16 01:41:48 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:48.024] RPC Daemon notified of new headers from=15542990 to=15542991 header sending=12.698µs log sending=383ns
Sep 16 01:41:48 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:48.024] [1/15 Headers] Waiting for Beacon Chain...
Sep 16 01:41:48 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:48.070] [1/15 Headers] Fork choice missing header with hash bb94645a8f48d078c62d1f190a2e64c56a2518d651ed78ebd6bd0ec786869333
Sep 16 01:41:48 erigon-1 erigon[526931]: erigon_1 | [INFO] [09-16|01:41:48.070] Commit cycle in=8.064µs
Also encountered runtime error running erigon 2022.08.3-alpha-9d88cb59 with teku/v22.9.0+4-ga9cb016.
Sep 15 19:34:59 hostname erigon[95233]: [INFO] [09-15|19:34:59.661] RPC Daemon notified of new headers from=15538239 to=15538240 header sending=9.74µs log sending=216ns
Sep 15 19:34:59 hostname erigon[95233]: [INFO] [09-15|19:34:59.661] [1/16 Headers] Waiting for Beacon Chain...
Sep 15 19:35:20 hostname erigon[95233]: [INFO] [09-15|19:35:20.307] [2/7 Bodies] Wrote block bodies block_num=15538239 delivery/sec=0B wasted/sec=4.1KB alloc=3.9GB sys=8.4GB
Sep 15 19:35:21 hostname erigon[95233]: [INFO] [09-15|19:35:21.834] [txpool] stat block=15538239 pending=2231 baseFee=30000 queued=29996 alloc=4.0GB sys=8.4GB
Sep 15 19:36:21 hostname erigon[95233]: [INFO] [09-15|19:36:21.401] [p2p] GoodPeers eth66=33
Sep 15 19:36:21 hostname erigon[95233]: [INFO] [09-15|19:36:21.833] [txpool] stat block=15538239 pending=3048 baseFee=30000 queued=30000 alloc=3.4GB sys=8.4GB
Sep 15 19:36:40 hostname erigon[95233]: [INFO] [09-15|19:36:40.307] [2/7 Bodies] DONE in=1m40.001048574s
Sep 15 19:36:40 hostname erigon[95233]: [INFO] [09-15|19:36:40.307] Timings (slower than 50ms) Bodies=1m40.001s
Sep 15 19:36:40 hostname erigon[95233]: [INFO] [09-15|19:36:40.307] Tables PlainState=0B AccountChangeSet=0B StorageChangeSet=0B BlockTransaction=0B TransactionLog=0B FreeList=0B
Sep 15 19:36:40 hostname erigon[95233]: [EROR] [09-15|19:36:40.312] Staged Sync err="runtime error: invalid memory address or nil pointer dereference, trace: [stageloop.go:129 panic.go:890 panic.go:260 signal_unix.go:835 fork_validator.go:279 fork_validator.go:216 stage_headers.go:673 collector.go:276 collector.go:144 stage_headers.go:686 stage_headers.go:732 stage_headers.go:195 st>
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.284] [1/16 Headers] Waiting for Beacon Chain...
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.285] [1/16 Headers] New payload missing parent
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.286] Commit cycle in=21.398µs
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.287] RPC Daemon notified of new headers from=15538239 to=15538240 header sending=30.301µs log sending=484ns
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.287] [1/16 Headers] Waiting for Beacon Chain...
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.288] [1/16 Headers] New payload missing parent
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.289] Commit cycle in=15.044µs
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.290] RPC Daemon notified of new headers from=15538239 to=15538240 header sending=29.952µs log sending=507ns
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.290] [1/16 Headers] Waiting for Beacon Chain...
Sep 15 19:36:42 hostname erigon[95233]: [INFO] [09-15|19:36:42.291] [1/16 Headers] New payload missing parent
Seeing the same with erigon (2022.08.03-alpha and 2022.09.01-alpha) + nimbus
As a workaround, Ii run this script every minute to restart erigon if it enters this state:
#!/bin/bash
set -eu
date
ERIGON_PID=$(systemctl show --property MainPID --value erigon.service)
OK=1
journalctl -u erigon.service -S "-1h" -g "invalid memory address" --output-fields _PID -o cat | grep -x $ERIGON_PID && OK=0
if [ $OK -ne 1 ]; then
echo "Erigon's dead, restarting"
systemctl restart erigon.service
else
echo "Erigon's healthy"
fi
@gitmewai Did you manage to fix the pending=10000 queued=10000 state in the txpool?
