erigon icon indicating copy to clipboard operation
erigon copied to clipboard

`erigon import` segmentation fault

Open yeonsookimdev opened this issue 1 year ago • 1 comments

System information

Erigon version: erigon version 2.59.0

OS & Version: Linux

Erigon Command (with flags/config): ./erigon import --datadir datadir.erigon 1-100000.chain.gz

Chain/Network: mainnet

Expected behaviour

Erigon imports the chain file and terminates without any error.

Actual behaviour

Segmentation fault between block 90,000 and 100,000.

Steps to reproduce the behaviour

  1. Sync Geth and export a chain file: ./geth export --datadir datadir.geth 1-100000.chain.gz 1 100000 (Download link: 1-100000.chain.gz)
  2. Import the chain file to Erigon with an empty datadir: ./erigon import --datadir datadir.erigon 1-100000.chain.gz

Backtrace

[INFO] [03-24|02:09:48.842] [2/15 Headers] Waiting for headers...    from=90000
[INFO] [03-24|02:09:48.886] [2/15 Headers] Processed                 highest=92500 age=8y8mo3w headers=2500 in=0.044 blk/sec=57269
[INFO] [03-24|02:09:48.892] [5/15 Bodies] Processing bodies...       from=90000 to=92500
[INFO] [03-24|02:09:49.420] [5/15 Bodies] Processed                  highest=92500 blocks=2500 in=0.528 blk/sec=4733
[INFO] [03-24|02:09:49.420] [6/15 Senders] Started                   from=90000 to=92500
[INFO] [03-24|02:09:49.451] [7/15 Execution] Blocks execution        from=90000 to=92500
[INFO] [03-24|02:09:49.570] [7/15 Execution] Completed on            block=92500
[INFO] [03-24|02:09:49.570] [8/15 HashState] Promoting plain state   from=90000 to=92500
[INFO] [03-24|02:09:49.570] [8/15 HashState] Incremental promotion   from=90000 to=92500 codes=true csbucket=AccountChangeSet
[INFO] [03-24|02:09:49.572] [8/15 HashState] Incremental promotion   from=90000 to=92500 codes=false csbucket=AccountChangeSet
[INFO] [03-24|02:09:49.579] [8/15 HashState] Incremental promotion   from=90000 to=92500 codes=false csbucket=StorageChangeSet
[INFO] [03-24|02:09:49.579] [9/15 IntermediateHashes] Generating intermediate hashes from=90000 to=92500
[INFO] [03-24|02:09:49.598] [10/15 CallTraces] Pruned call trace intermediate table from=1 to=2499
[INFO] [03-24|02:09:49.609] [13/15 LogIndex] processing              from=90001 to=92500
[INFO] [03-24|02:09:49.626] RPC Daemon notified of new headers       from=91476 to=92500 amount=1024 hash=0x0b01b70e67cb0659af8e0b76a37f12c08918f052e5b29a48efc032035cdd7a3a header sending=803.614µs log sending=205ns
[INFO] [03-24|02:09:49.626] Timings (slower than 50ms)               Bodies=528ms Execution=118ms
[INFO] [03-24|02:09:49.626] Tables                                   PlainState=1.1MB AccountChangeSet=7.1MB StorageChangeSet=392.0KB BlockTransaction=3.4MB TransactionLog=96.0KB FreeList=16.0KB ReclaimableSpace=32.0MB
[INFO] [03-24|02:09:49.774] [2/15 Headers] Waiting for headers...    from=92500
[INFO] [03-24|02:09:49.800] [2/15 Headers] Processed                 highest=95000 age=8y8mo3w headers=2500 in=0.025 blk/sec=98075
[INFO] [03-24|02:09:49.804] [5/15 Bodies] Processing bodies...       from=92500 to=95000
[INFO] [03-24|02:09:50.228] [5/15 Bodies] Processed                  highest=95000 blocks=2500 in=0.423 blk/sec=5905
[INFO] [03-24|02:09:50.228] [6/15 Senders] Started                   from=92500 to=95000
[INFO] [03-24|02:09:50.261] [7/15 Execution] Blocks execution        from=92500 to=95000
[INFO] [03-24|02:09:50.407] [7/15 Execution] Completed on            block=95000
[INFO] [03-24|02:09:50.407] [8/15 HashState] Promoting plain state   from=92500 to=95000
[INFO] [03-24|02:09:50.407] [8/15 HashState] Incremental promotion   from=92500 to=95000 codes=true csbucket=AccountChangeSet
[INFO] [03-24|02:09:50.410] [8/15 HashState] Incremental promotion   from=92500 to=95000 codes=false csbucket=AccountChangeSet
[INFO] [03-24|02:09:50.419] [8/15 HashState] Incremental promotion   from=92500 to=95000 codes=false csbucket=StorageChangeSet
[INFO] [03-24|02:09:50.419] [9/15 IntermediateHashes] Generating intermediate hashes from=92500 to=95000
[INFO] [03-24|02:09:50.445] [10/15 CallTraces] Pruned call trace intermediate table from=2500 to=4999
[INFO] [03-24|02:09:50.461] [13/15 LogIndex] processing              from=92501 to=95000
[INFO] [03-24|02:09:50.482] RPC Daemon notified of new headers       from=93976 to=95000 amount=1024 hash=0xd47fce4308718e7d2f68f6641395d424e1e8d08e82774fb5d693e6bc49a30db6 header sending=798.159µs log sending=187ns
[INFO] [03-24|02:09:50.482] Timings (slower than 50ms)               Bodies=423ms Execution=146ms
[INFO] [03-24|02:09:50.482] Tables                                   PlainState=1.1MB AccountChangeSet=7.4MB StorageChangeSet=392.0KB BlockTransaction=3.6MB TransactionLog=104.0KB FreeList=16.0KB ReclaimableSpace=32.0MB
[INFO] [03-24|02:09:50.629] [2/15 Headers] Waiting for headers...    from=95000
[INFO] [03-24|02:09:50.657] [2/15 Headers] Processed                 highest=97500 age=8y8mo3w headers=2500 in=0.028 blk/sec=88652
[INFO] [03-24|02:09:50.662] [5/15 Bodies] Processing bodies...       from=95000 to=97500
[INFO] [03-24|02:09:51.040] [snapshots:blocks:retire] Blocks Stat    blocks=0k indices=0k alloc=634.9MB sys=896.1MB
[INFO] [03-24|02:09:51.339] [5/15 Bodies] Processed                  highest=97500 blocks=2500 in=0.677 blk/sec=3695
[INFO] [03-24|02:09:51.339] [6/15 Senders] Started                   from=95000 to=97500
[INFO] [03-24|02:09:51.357] [7/15 Execution] Blocks execution        from=95000 to=97500
[INFO] [03-24|02:09:51.433] [7/15 Execution] Completed on            block=97500
[INFO] [03-24|02:09:51.433] [8/15 HashState] Promoting plain state   from=95000 to=97500
[INFO] [03-24|02:09:51.433] [8/15 HashState] Incremental promotion   from=95000 to=97500 codes=true csbucket=AccountChangeSet
[INFO] [03-24|02:09:51.436] [8/15 HashState] Incremental promotion   from=95000 to=97500 codes=false csbucket=AccountChangeSet
[INFO] [03-24|02:09:51.449] [8/15 HashState] Incremental promotion   from=95000 to=97500 codes=false csbucket=StorageChangeSet
[INFO] [03-24|02:09:51.455] [9/15 IntermediateHashes] Generating intermediate hashes from=95000 to=97500
[INFO] [03-24|02:09:51.482] [10/15 CallTraces] Pruned call trace intermediate table from=5000 to=7499
[INFO] [03-24|02:09:51.493] [13/15 LogIndex] processing              from=95001 to=97500
[INFO] [03-24|02:09:51.513] RPC Daemon notified of new headers       from=96476 to=97500 amount=1024 hash=0x2b6e650b0039efb04a1f2d3d0686af7ecaf6efbbea58ab6592c186e0b35eb5eb header sending=902.711µs log sending=272ns
[INFO] [03-24|02:09:51.513] Timings (slower than 50ms)               Bodies=676ms Execution=76ms
[INFO] [03-24|02:09:51.513] Tables                                   PlainState=1.2MB AccountChangeSet=7.6MB StorageChangeSet=400.0KB BlockTransaction=3.8MB TransactionLog=104.0KB FreeList=16.0KB ReclaimableSpace=32.0MB
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x198 pc=0x16bde5c]

goroutine 372398 [running]:
github.com/ledgerwatch/erigon-lib/recsplit.(*Index).BaseDataID(...)
        github.com/ledgerwatch/[email protected]/recsplit/index.go:238
github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks.(*BlockReader).FirstTxnNumNotInSnapshots(0xc0033ae920)
        github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks/block_reader.go:925 +0x13c
github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks.DumpBlocks({0x308a948, 0xc00072c500}, 0x0, 0x3e8, 0xc000c16640, {0xc00005b440, 0x35}, {0xc00005b480, 0x3a}, {0x30967d0, ...}, ...)
        github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:1400 +0x7b
github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks.(*BlockRetire).retireBlocks(0xc002e9c580, {0x308a948, 0xc00072c500}, 0x0, 0x17318, 0x4, 0xc026d18140, 0xc026d18160)
        github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:1223 +0x3bc
github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks.(*BlockRetire).RetireBlocks(0xc002e9c580, {0x308a948, 0xc00072c500}, 0x0, 0x17318, 0x4, 0xc026d18140, 0xc026d18160)
        github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:1337 +0x115
github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks.(*BlockRetire).RetireBlocksInBackground.func1()
        github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:1312 +0xda
created by github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks.(*BlockRetire).RetireBlocksInBackground in goroutine 1
        github.com/ledgerwatch/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:1305 +0xfa

yeonsookimdev avatar Mar 23 '24 17:03 yeonsookimdev

2.58.2 panics at the beginning of the erigon import with segmentation fault. (Download link: erigon-2.58.2-import.log)

[INFO] [03-24|02:20:40.625] logging to file system                   log dir=datadir.erigon/logs file prefix=erigon log level=info json=false
[INFO] [03-24|02:20:40.625] Starting Erigon on Ethereum mainnet... 
[INFO] [03-24|02:20:40.627] Maximum peer count                       ETH=100 total=100
[INFO] [03-24|02:20:40.627] starting HTTP APIs                       port=8545 APIs=eth,erigon,engine
[INFO] [03-24|02:20:40.627] torrent verbosity                        level=WRN
[INFO] [03-24|02:20:42.731] Set global gas cap                       cap=50000000
[INFO] [03-24|02:20:42.767] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
[INFO] [03-24|02:20:42.767] Opening Database                         label=chaindata path=/scratch/alkorang/ethereum-erigon/datadir.erigon/chaindata
[INFO] [03-24|02:20:42.769] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
[INFO] [03-24|02:20:42.769] Re-Opening DB in exclusive mode to apply migrations 
[INFO] [03-24|02:20:42.771] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
[INFO] [03-24|02:20:42.772] Apply migration                          name=db_schema_version5
[INFO] [03-24|02:20:42.772] Applied migration                        name=db_schema_version5
[INFO] [03-24|02:20:42.772] Apply migration                          name=txs_begin_end
[INFO] [03-24|02:20:42.772] Applied migration                        name=txs_begin_end
[INFO] [03-24|02:20:42.772] Apply migration                          name=txs_v3
[INFO] [03-24|02:20:42.772] Applied migration                        name=txs_v3
[INFO] [03-24|02:20:42.772] Apply migration                          name=prohibit_new_downloads_lock
[INFO] [03-24|02:20:42.772] Applied migration                        name=prohibit_new_downloads_lock
[INFO] [03-24|02:20:42.772] Updated DB schema to                     version=6.1.0
[INFO] [03-24|02:20:42.774] [db] open                                lable=chaindata sizeLimit=12TB pageSize=8192
[INFO] [03-24|02:20:42.892] Writing custom genesis block             hash=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [03-24|02:20:42.893] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: 1710338135, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
[INFO] [03-24|02:20:42.921] Initialising Ethereum protocol           network=1
[INFO] [03-24|02:20:42.921] Disk storage enabled for ethash DAGs     dir=/scratch/alkorang/ethereum-erigon/datadir.erigon/ethash-dags count=2
[INFO] [03-24|02:20:42.923] Starting private RPC server              on=127.0.0.1:9090
[INFO] [03-24|02:20:42.923] new subscription to logs established 
[INFO] [03-24|02:20:42.923] rpc filters: subscribing to Erigon events 
[INFO] [03-24|02:20:42.923] [txpool] Started 
[INFO] [03-24|02:20:42.923] New txs subscriber joined 
[INFO] [03-24|02:20:42.923] new subscription to newHeaders established 
[INFO] [03-24|02:20:42.923] Importing blockchain                     file=1-100000.chain.gz
[INFO] [03-24|02:20:42.924] Reading JWT secret                       path=/scratch/alkorang/ethereum-erigon/datadir.erigon/jwt.hex
[INFO] [03-24|02:20:42.924] Generated JWT secret                     path=/scratch/alkorang/ethereum-erigon/datadir.erigon/jwt.hex
SIGSEGV: segmentation violation
PC=0x7f224c102bd0 m=24 sigcode=1
signal arrived during cgo execution

goroutine 2692 [syscall]:
runtime.cgocall(0x20ef8b0, 0xc003344588)
	/usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc003344560 sp=0xc003344528 pc=0x42596b
net._C2func_getaddrinfo(0xc0033b07e0, 0x0, 0xc001eeda10, 0xc000b8cb58)
	_cgo_gotypes.go:100 +0x55 fp=0xc003344588 sp=0xc003344560 pc=0x623235
net._C_getaddrinfo.func1(0x42eac5?, 0x8?, 0x2316560?, 0x1?)
	/usr/local/go/src/net/cgo_unix_cgo.go:78 +0x7a fp=0xc0033445f0 sp=0xc003344588 pc=0x623bba
net._C_getaddrinfo(0xc0019f00c6?, 0x9?, 0x30?, 0x7ffd7f48ca85?)
	/usr/local/go/src/net/cgo_unix_cgo.go:78 +0x13 fp=0xc003344620 sp=0xc0033445f0 pc=0x623af3
net.cgoLookupHostIP({0xc0019f00c0, 0x3}, {0xc0019f00c6, 0x9})
	/usr/local/go/src/net/cgo_unix.go:166 +0x24f fp=0xc003344760 sp=0xc003344620 pc=0x5ee1ef
net.cgoLookupIP.func1()
	/usr/local/go/src/net/cgo_unix.go:215 +0x25 fp=0xc003344790 sp=0xc003344760 pc=0x5ee905
net.doBlockingWithCtx[...].func1()
	/usr/local/go/src/net/cgo_unix.go:56 +0x35 fp=0xc0033447e0 sp=0xc003344790 pc=0x623f95
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0033447e8 sp=0xc0033447e0 pc=0x490421
created by net.doBlockingWithCtx[...] in goroutine 2691
	/usr/local/go/src/net/cgo_unix.go:54 +0xd8

...

yeonsookimdev avatar Mar 23 '24 17:03 yeonsookimdev

I tested erigon import with past versions. Until version 2.52.5 it works fine.

It immediately panics from version 2.53.0.

INFO[05-09|21:43:56.496] logging to file system                   log dir=datadir.erigon-2.53/logs file prefix=erigon log level=info json=false
INFO[05-09|21:43:56.496] Starting Erigon on Ethereum mainnet... 
INFO[05-09|21:43:56.497] Maximum peer count                       ETH=100 total=100
INFO[05-09|21:43:56.498] starting HTTP APIs                       APIs=eth,erigon,engine
INFO[05-09|21:43:56.498] torrent verbosity                        level=WRN
INFO[05-09|21:44:04.603] Set global gas cap                       cap=50000000
INFO[05-09|21:44:04.647] [Downloader] Runnning with               ipv6-enabled=true ipv4-enabled=true download.rate=16mb upload.rate=4mb
INFO[05-09|21:44:04.647] Opening Database                         label=chaindata path=/scratch/alkorang/ethereum-erigon/datadir.erigon-2.53/chaindata
INFO[05-09|21:44:04.648] [db] chaindata                           sizeLimit=3TB pageSize=8192
INFO[05-09|21:44:04.649] Initialised chain configuration          config="{ChainID: 1, Homestead: 1150000, DAO: 1920000, Tangerine Whistle: 2463000, Spurious Dragon: 2675000, Byzantium: 4370000, Constantinople: 7280000, Petersburg: 7280000, Istanbul: 9069000, Muir Glacier: 9200000, Berlin: 12244000, London: 12965000, Arrow Glacier: 13773000, Gray Glacier: 15050000, Terminal Total Difficulty: 58750000000000000000000, Merge Netsplit: <nil>, Shanghai: 1681338455, Cancun: <nil>, Prague: <nil>, Engine: ethash}" genesis=0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3
WARN[05-09|21:44:04.653] bind protocol to port has failed: port is busy protocols="eth/[68 67]" port=30303
INFO[05-09|21:44:04.653] Initialising Ethereum protocol           network=1
INFO[05-09|21:44:04.653] Disk storage enabled for ethash DAGs     dir=/scratch/alkorang/ethereum-erigon/datadir.erigon-2.53/ethash-dags count=2
0
INFO[05-09|21:44:04.660] Starting private RPC server              on=127.0.0.1:9090
INFO[05-09|21:44:04.660] new subscription to logs established 
INFO[05-09|21:44:04.660] rpc filters: subscribing to Erigon events 
INFO[05-09|21:44:04.660] new subscription to newHeaders established 
INFO[05-09|21:44:04.660] New txs subscriber joined 
INFO[05-09|21:44:04.660] Importing blockchain                     file=1-100000.chain.gz
INFO[05-09|21:44:04.661] Reading JWT secret                       path=/scratch/alkorang/ethereum-erigon/datadir.erigon-2.53/jwt.hex
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x63 pc=0x7efe79a2d298]

runtime stack:
runtime.throw({0x2b182e3?, 0x7efe9a8d8dd0?})
	/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7efe9a8d8820 sp=0x7efe9a8d87f0 pc=0x45757d
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7efe9a8d8880 sp=0x7efe9a8d8820 pc=0x46eaa9

goroutine 28 [syscall]:
runtime.cgocall(0x21c5440, 0xc00089bd58)
	/usr/local/go/src/runtime/cgocall.go:157 +0x5c fp=0xc00089bd30 sp=0xc00089bcf8 pc=0x4224bc
net._C2func_getaddrinfo(0xc000631a40, 0x0, 0xc00023dc50, 0xc000126900)
	_cgo_gotypes.go:99 +0x56 fp=0xc00089bd58 sp=0xc00089bd30 pc=0x636bb6
net._C_getaddrinfo.func1(0x42c2e7?, 0x8?, 0x242bda0?, 0xc000631a01?)
	/usr/local/go/src/net/cgo_unix_cgo.go:74 +0x85 fp=0xc00089bdc0 sp=0xc00089bd58 pc=0x637785
net._C_getaddrinfo(0xc000101400?, 0xc001bca030?, 0x0?, 0xc000631660?)
	/usr/local/go/src/net/cgo_unix_cgo.go:74 +0x19 fp=0xc00089bdf0 sp=0xc00089bdc0 pc=0x6376b9
net.cgoLookupIPCNAME({0x293cfe6, 0x3}, {0xc000631660, 0x9})
	/usr/local/go/src/net/cgo_unix.go:153 +0x175 fp=0xc00089bf38 sp=0xc00089bdf0 pc=0x5feed5
net.cgoIPLookup(0x49c0390?, {0x293cfe6?, 0xc000631a30?}, {0xc000631660?, 0xc000caa5a0?})
	/usr/local/go/src/net/cgo_unix.go:210 +0x3b fp=0xc00089bfa8 sp=0xc00089bf38 pc=0x5ff59b
net.cgoLookupIP.func1()
	/usr/local/go/src/net/cgo_unix.go:220 +0x36 fp=0xc00089bfe0 sp=0xc00089bfa8 pc=0x5ff9d6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00089bfe8 sp=0xc00089bfe0 pc=0x48ed21
created by net.cgoLookupIP
	/usr/local/go/src/net/cgo_unix.go:220 +0x12a

...

yeonsookimdev avatar May 09 '24 12:05 yeonsookimdev

@taratorio could you take a look at this one plz

yperbasis avatar May 31 '24 12:05 yperbasis

@alkorang this works on v2.60.1. Closing this issue but feel free to re-open if the issue surfaces again.

taratorio avatar Jun 04 '24 17:06 taratorio