go-ethereum
go-ethereum copied to clipboard
Published
20 hours ago •
ethereum
ethereum
`Imported new potential chain segment` poor performance when `ancient` is on external drive — v1.13.5
System information
Geth version:
$ geth --version
geth version 1.13.5-stable-916d6a44
CL client & version:
$ lighthouse --version
Lighthouse v4.5.0-441fc16
BLS library: blst
SHA256 hardware acceleration: true
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (false)
OS & Version:
$ uname -a
Linux nuc 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Expected behaviour
Typical high-performant attestation behavior.
Actual behaviour
Much higher than usual instances of:
- missed attestations
- attestations being included in late blocks
- attestations not voting on the correct chain head
Steps to reproduce the behaviour
Run geth on version 1.13.5
Startup command
[Unit]
Description=Geth Execution Client
Wants=network-online.target
After=network-online.target
[Service]
User=geth
Group=geth
Type=simple
Restart=always
ExecStart=/usr/local/bin/geth \
--datadir=/var/lib/geth \
--port=30303 \
--http=true \
--http.port=8545 \
--authrpc.jwtsecret=/var/lib/ethereum-shared/jwt-primary.hex \
--authrpc.port=8551 \
--db.engine=pebble \
--state.scheme=path \
[Install]
WantedBy=multi-user.target
Backtrace
INFO [11-15|20:15:36.780] Imported new potential chain segment number=18,579,606 hash=dbab5c..37f2c0 blocks=1 txs=185 mgas=16.103 elapsed=150.738ms mgasps=106.828 snapdiffs=5.99MiB triediffs=178.01MiB triedirty=12.25MiB
INFO [11-15|20:15:36.910] Chain head was updated number=18,579,606 hash=dbab5c..37f2c0 root=29d20e..e2074f elapsed=2.483595ms
INFO [11-15|20:15:49.853] Imported new potential chain segment number=18,579,607 hash=f712d1..aa7e0a blocks=1 txs=137 mgas=15.814 elapsed=1.138s mgasps=13.892 snapdiffs=6.04MiB triediffs=178.04MiB triedirty=12.82MiB
INFO [11-15|20:15:49.992] Chain head was updated number=18,579,607 hash=f712d1..aa7e0a root=85bf19..16d40f elapsed=3.660048ms
INFO [11-15|20:16:02.419] Imported new potential chain segment number=18,579,608 hash=ecba4e..265465 blocks=1 txs=126 mgas=13.707 elapsed=1.984s mgasps=6.905 snapdiffs=6.07MiB triediffs=177.81MiB triedirty=13.24MiB
INFO [11-15|20:16:02.592] Chain head was updated number=18,579,608 hash=ecba4e..265465 root=0aadca..cab0b8 elapsed=3.509709ms
INFO [11-15|20:16:16.972] Imported new potential chain segment number=18,579,609 hash=d92259..820503 blocks=1 txs=214 mgas=22.833 elapsed=2.050s mgasps=11.135 snapdiffs=6.11MiB triediffs=178.43MiB triedirty=13.67MiB
INFO [11-15|20:16:17.131] Chain head was updated number=18,579,609 hash=d92259..820503 root=69fa1b..8cc8ed elapsed=3.917986ms
INFO [11-15|20:16:26.642] Imported new potential chain segment number=18,579,610 hash=66882a..6c500f blocks=1 txs=117 mgas=10.856 elapsed=2.011s mgasps=5.397 snapdiffs=6.15MiB triediffs=178.37MiB triedirty=14.08MiB
INFO [11-15|20:16:26.776] Chain head was updated number=18,579,610 hash=66882a..6c500f root=6c299b..ccd5ff elapsed=2.88514ms
INFO [11-15|20:16:38.755] Imported new potential chain segment number=18,579,611 hash=972d8c..06f8dd blocks=1 txs=141 mgas=10.532 elapsed=1.931s mgasps=5.453 snapdiffs=6.18MiB triediffs=178.29MiB triedirty=14.47MiB
INFO [11-15|20:16:38.947] Chain head was updated number=18,579,611 hash=972d8c..06f8dd root=fd3463..1c5048 elapsed=2.612727ms
INFO [11-15|20:16:50.525] Imported new potential chain segment number=18,579,612 hash=2f1817..fbc1b5 blocks=1 txs=188 mgas=13.286 elapsed=2.107s mgasps=6.305 snapdiffs=6.22MiB triediffs=178.40MiB triedirty=15.02MiB
INFO [11-15|20:16:50.663] Chain head was updated number=18,579,612 hash=2f1817..fbc1b5 root=bec5ce..3c5e22 elapsed=3.444437ms
INFO [11-15|20:17:01.373] Imported new potential chain segment number=18,579,613 hash=037f6b..9a1169 blocks=1 txs=135 mgas=11.636 elapsed=933.221ms mgasps=12.469 snapdiffs=6.26MiB triediffs=178.21MiB triedirty=15.55MiB
INFO [11-15|20:17:01.509] Chain head was updated number=18,579,613 hash=037f6b..9a1169 root=255985..95ccee elapsed=2.088793ms
INFO [11-15|20:17:12.655] Imported new potential chain segment number=18,579,614 hash=689b77..3533fa blocks=1 txs=216 mgas=16.396 elapsed=184.686ms mgasps=88.777 snapdiffs=6.30MiB triediffs=178.65MiB triedirty=15.97MiB
INFO [11-15|20:17:12.772] Chain head was updated number=18,579,614 hash=689b77..3533fa root=081983..580735 elapsed=3.296078ms
INFO [11-15|20:17:25.151] Imported new potential chain segment number=18,579,615 hash=0df623..47a13a blocks=1 txs=138 mgas=14.496 elapsed=189.998ms mgasps=76.296 snapdiffs=6.34MiB triediffs=178.73MiB triedirty=16.38MiB
INFO [11-15|20:17:25.285] Chain head was updated number=18,579,615 hash=0df623..47a13a root=6e09f5..49b456 elapsed=4.620928ms
INFO [11-15|20:17:38.091] Imported new potential chain segment number=18,579,616 hash=78bc61..c2c488 blocks=1 txs=206 mgas=18.955 elapsed=204.642ms mgasps=92.626 snapdiffs=6.39MiB triediffs=179.20MiB triedirty=16.74MiB
INFO [11-15|20:17:38.212] Chain head was updated number=18,579,616 hash=78bc61..c2c488 root=6ebe99..9df013 elapsed=2.753035ms
INFO [11-15|20:17:51.712] Imported new potential chain segment number=18,579,617 hash=2cf10c..a8c0a9 blocks=1 txs=145 mgas=14.961 elapsed=1.137s mgasps=13.148 snapdiffs=6.43MiB triediffs=179.38MiB triedirty=17.16MiB
INFO [11-15|20:17:51.861] Chain head was updated number=18,579,617 hash=2cf10c..a8c0a9 root=cbf2f6..079f30 elapsed=4.30093ms
INFO [11-15|20:18:04.540] Imported new potential chain segment number=18,579,618 hash=c4b4b0..dbc556 blocks=1 txs=152 mgas=23.963 elapsed=2.023s mgasps=11.844 snapdiffs=6.47MiB triediffs=179.80MiB triedirty=17.44MiB
INFO [11-15|20:18:04.670] Chain head was updated number=18,579,618 hash=c4b4b0..dbc556 root=4d75ec..b2f8e8 elapsed=5.809166ms
INFO [11-15|20:18:14.677] Imported new potential chain segment number=18,579,619 hash=915087..1d32d9 blocks=1 txs=149 mgas=14.145 elapsed=2.041s mgasps=6.928 snapdiffs=6.50MiB triediffs=179.93MiB triedirty=17.81MiB
INFO [11-15|20:18:14.792] Chain head was updated number=18,579,619 hash=915087..1d32d9 root=c87184..be711b elapsed=3.238122ms
INFO [11-15|20:18:26.486] Imported new potential chain segment number=18,579,620 hash=f14a75..03ced5 blocks=1 txs=254 mgas=15.461 elapsed=1.994s mgasps=7.750 snapdiffs=6.54MiB triediffs=180.31MiB triedirty=18.14MiB
INFO [11-15|20:18:26.602] Chain head was updated number=18,579,620 hash=f14a75..03ced5 root=ee9481..28f946 elapsed=3.521982ms
Looking at the logs, it seems like there are periods where the elapsed time on "Imported new potential chain segment" logs are on the order of seconds. These periods last for a few slots, then go back to "normal" processing times (on the order of 1-200ms) for a few slots, then repeat.
I suspect that it's these periods of "slow" chain segmet processing times that are leading to poor attestation performance, but that's an assumption. That is the only difference I can see between the logs of 1.13.5, vs 1.13.4 (apart from no more "Peer delivering stale transaction" logs!)
This slow chain segment processing issue (assuming it's the culprit for poor attestation performance) was not occurring on my machine when running version 1.13.4.
Thanks for reporting, could you try upping the log level for the trie debug.vmodule("trie=5") in the console and post more logs? Do you have a dashboard by chance where we could see some more numbers?
I think it might be the freezer fsync or the trie flush
@MariusVanDerWijden I don't have a dashboard... if it's worthwhile, can you point me in a direction to set one up?
Also, I'm unsure how/where to use debug.vmodule("trie=5"). Is this something I can do as a startup flag? I'm currently viewing logs via journalctl. Apologies for my noobness here.
edit:
As per the geth --help output:
--log.vmodule value ($GETH_LOG_VMODULE)
Per-module verbosity: comma-separated list of <pattern>=<level> (e.g. eth/*=5,p2p=4)
I'm running geth via systemd with a --log.vmodule trie=5 flag. It's not giving any more logs than normal, though. I made sure that other log.vmodule flags did show more logs (eth/*=6 immediately started spitting out a ton of logs I've never seen before). Are you sure that trie=5 is right?
Ah maybe "trie/triedb/pathdb=5" or "trie/*=5". You can connect a console to your running geth node, so you don't need to restart to change the flags with geth attach ~/path/to/geth.ipc console
It would be interesting to see if any log.Debug("Pruned state history", "items", pruned, "tailid", oldest) show up
Great, thanks for your help walking me through this @MariusVanDerWijden! I've got those trie debug logs showing up. I'll let this run for ~10 minutes then dump the logs here.
INFO [11-16|13:36:00.646] Starting Geth on Ethereum mainnet...
INFO [11-16|13:36:00.647] Bumping default cache on mainnet provided=1024 updated=4096
INFO [11-16|13:36:00.648] Maximum peer count ETH=50 LES=0 total=50
INFO [11-16|13:36:00.649] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [11-16|13:36:00.649] Using pebble as db engine
INFO [11-16|13:36:00.651] Set global gas cap cap=50,000,000
INFO [11-16|13:36:00.651] Initializing the KZG library backend=gokzg
INFO [11-16|13:36:00.677] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB
INFO [11-16|13:36:00.746] Using pebble as the backing database
INFO [11-16|13:36:00.746] Allocated cache and file handles database=/var/lib/geth/geth/chaindata cache=2.00GiB handles=262,144
INFO [11-16|13:36:04.186] Found legacy ancient chain path location=/var/lib/geth/geth/chaindata/ancient
INFO [11-16|13:36:04.212] Opened ancient database database=/var/lib/geth/geth/chaindata/ancient readonly=false
INFO [11-16|13:36:04.215] State scheme set by user scheme=path
INFO [11-16|13:36:04.222] Initialising Ethereum protocol network=1 dbversion=8
WARN [11-16|13:36:04.222] Sanitizing invalid node buffer size provided=1024.00MiB updated=256.00MiB
INFO [11-16|13:36:05.788] Opened ancient database database=/var/lib/geth/geth/chaindata/ancient/state readonly=false
WARN [11-16|13:36:05.791] Path-based state scheme is an experimental feature
INFO [11-16|13:36:05.791]
INFO [11-16|13:36:05.791] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [11-16|13:36:05.792] Chain ID: 1 (mainnet)
INFO [11-16|13:36:05.792] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [11-16|13:36:05.792]
INFO [11-16|13:36:05.792] Pre-Merge hard forks (block based):
INFO [11-16|13:36:05.792] - Homestead: #1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [11-16|13:36:05.792] - DAO Fork: #1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [11-16|13:36:05.792] - Tangerine Whistle (EIP 150): #2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [11-16|13:36:05.792] - Spurious Dragon/1 (EIP 155): #2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [11-16|13:36:05.792] - Spurious Dragon/2 (EIP 158): #2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [11-16|13:36:05.792] - Byzantium: #4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [11-16|13:36:05.792] - Constantinople: #7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [11-16|13:36:05.792] - Petersburg: #7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [11-16|13:36:05.792] - Istanbul: #9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [11-16|13:36:05.792] - Muir Glacier: #9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [11-16|13:36:05.792] - Berlin: #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [11-16|13:36:05.792] - London: #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [11-16|13:36:05.792] - Arrow Glacier: #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [11-16|13:36:05.792] - Gray Glacier: #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [11-16|13:36:05.792]
INFO [11-16|13:36:05.792] Merge configured:
INFO [11-16|13:36:05.792] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [11-16|13:36:05.792] - Network known to be merged: true
INFO [11-16|13:36:05.792] - Total terminal difficulty: 58750000000000000000000
INFO [11-16|13:36:05.792]
INFO [11-16|13:36:05.792] Post-Merge hard forks (timestamp based):
INFO [11-16|13:36:05.792] - Shanghai: @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [11-16|13:36:05.792]
INFO [11-16|13:36:05.792] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [11-16|13:36:05.792]
INFO [11-16|13:36:05.793] Loaded most recent local block number=18,584,782 hash=5f6023..103ed6 td=58,750,003,716,598,352,816,469 age=18s
INFO [11-16|13:36:05.793] Loaded most recent local finalized block number=18,584,689 hash=e7ccb4..5f4bf5 td=58,750,003,716,598,352,816,469 age=18m54s
INFO [11-16|13:36:05.793] Loaded last snap-sync pivot marker number=18,180,334
INFO [11-16|13:36:06.173] Initialized transaction indexer limit=2,350,000
INFO [11-16|13:36:06.173] Loaded local transaction journal transactions=0 dropped=0
INFO [11-16|13:36:06.173] Regenerated local transaction journal transactions=0 accounts=0
WARN [11-16|13:36:06.177] Switch sync mode from snap sync to full sync reason="snap sync complete"
INFO [11-16|13:36:06.177] Chain post-merge, sync via beacon client
INFO [11-16|13:36:06.177] Gasprice oracle is ignoring threshold set threshold=2
WARN [11-16|13:36:06.179] Engine API enabled protocol=eth
INFO [11-16|13:36:06.179] Starting peer-to-peer node instance=Geth/v1.13.5-stable-916d6a44/linux-amd64/go1.20.2
INFO [11-16|13:36:06.196] New local node record seq=1,681,344,381,526 id=a0146bc537df9cf1 ip=127.0.0.1 udp=30303 tcp=30303
INFO [11-16|13:36:06.198] IPC endpoint opened url=/var/lib/geth/geth.ipc
INFO [11-16|13:36:06.198] Loaded JWT secret file path=/var/lib/ethereum-shared/jwt-primary.hex crc32=0x1b524a01
INFO [11-16|13:36:06.198] HTTP server started endpoint=127.0.0.1:8545 auth=false prefix= cors= vhosts=localhost
INFO [11-16|13:36:06.198] WebSocket enabled url=ws://127.0.0.1:8551
INFO [11-16|13:36:06.198] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost
INFO [11-16|13:36:06.203] Started P2P networking self=enode://2acdf0bc4b77479c8f0b3dcfd22e754e977a969317d4dc676b25efdd0ccc217a671deb2eb9dc491cc7ec7ec31857e2745e6a772c80ffc31f0b4ad4db7a111547@127.0.0.1:30303
INFO [11-16|13:36:08.364] New local node record seq=1,681,344,381,527 id=a0146bc537df9cf1 ip=174.61.110.113 udp=30303 tcp=30303
INFO [11-16|13:36:08.840] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [11-16|13:36:09.292] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [11-16|13:36:10.560] Imported new potential chain segment number=18,584,783 hash=d8ab97..bd8e12 blocks=1 txs=179 mgas=17.724 elapsed=328.003ms mgasps=54.035 snapdiffs=6.37MiB triediffs=180.72MiB triedirty=51.75MiB
INFO [11-16|13:36:10.680] Chain head was updated number=18,584,783 hash=d8ab97..bd8e12 root=a7a0c7..586acd elapsed=4.47569ms
INFO [11-16|13:36:13.944] Imported new potential chain segment number=18,584,784 hash=97283a..c1f468 blocks=1 txs=213 mgas=16.708 elapsed=330.491ms mgasps=50.555 snapdiffs=6.42MiB triediffs=181.50MiB triedirty=51.95MiB
INFO [11-16|13:36:14.042] Chain head was updated number=18,584,784 hash=97283a..c1f468 root=f284b7..a5f5f2 elapsed=2.929812ms
INFO [11-16|13:36:16.380] Looking for peers peercount=1 tried=82 static=0
INFO [11-16|13:36:24.818] Imported new potential chain segment number=18,584,785 hash=9adcc5..1b6fdf blocks=1 txs=140 mgas=12.461 elapsed=298.336ms mgasps=41.770 snapdiffs=6.46MiB triediffs=181.15MiB triedirty=52.30MiB
INFO [11-16|13:36:26.355] Chain head was updated number=18,584,785 hash=9adcc5..1b6fdf root=d8ae03..70f8e9 elapsed=2.485708ms
INFO [11-16|13:36:26.413] Looking for peers peercount=1 tried=145 static=0
INFO [11-16|13:36:36.449] Looking for peers peercount=3 tried=98 static=0
INFO [11-16|13:36:37.770] Imported new potential chain segment number=18,584,786 hash=7c3a82..38c554 blocks=1 txs=152 mgas=14.595 elapsed=174.314ms mgasps=83.728 snapdiffs=6.50MiB triediffs=181.43MiB triedirty=52.51MiB
INFO [11-16|13:36:37.888] Chain head was updated number=18,584,786 hash=7c3a82..38c554 root=48639a..4c5c6a elapsed=3.1817ms
INFO [11-16|13:36:50.316] Imported new potential chain segment number=18,584,787 hash=77a22c..121b5c blocks=1 txs=192 mgas=16.305 elapsed=194.992ms mgasps=83.620 snapdiffs=6.55MiB triediffs=181.23MiB triedirty=53.07MiB
INFO [11-16|13:36:50.440] Chain head was updated number=18,584,787 hash=77a22c..121b5c root=4958ff..54f7db elapsed=2.920029ms
INFO [11-16|13:36:56.480] Looking for peers peercount=3 tried=40 static=0
INFO [11-16|13:37:01.511] Imported new potential chain segment number=18,584,788 hash=e59761..5f9f3e blocks=1 txs=105 mgas=14.536 elapsed=1.718s mgasps=8.457 snapdiffs=6.58MiB triediffs=181.01MiB triedirty=53.40MiB
INFO [11-16|13:37:01.651] Chain head was updated number=18,584,788 hash=e59761..5f9f3e root=fbe5fe..28727d elapsed=2.588535ms
WARN [11-16|13:37:06.730] Served eth_coinbase reqid=3 duration="25.875µs" err="etherbase must be explicitly specified"
INFO [11-16|13:37:14.626] Imported new potential chain segment number=18,584,789 hash=654917..f36736 blocks=1 txs=148 mgas=15.025 elapsed=2.116s mgasps=7.098 snapdiffs=6.62MiB triediffs=181.39MiB triedirty=53.63MiB
INFO [11-16|13:37:14.779] Chain head was updated number=18,584,789 hash=654917..f36736 root=b79848..5f2af6 elapsed=3.729253ms
[turned on trie debug logging here via debug.vmodule("trie/*=5")]
DEBUG[11-16|13:37:25.309] Created new diff layer id=404,456 block=18,584,790 nodes=3522 size=1.31MiB
DEBUG[11-16|13:37:27.221] Stored state history id=404,328 block=18,584,662 data=12.00KiB index=18.94KiB elapsed=1.910s
DEBUG[11-16|13:37:27.223] Pruned state history items=1 tailid=314,329
INFO [11-16|13:37:27.223] Imported new potential chain segment number=18,584,790 hash=0031ef..25e829 blocks=1 txs=134 mgas=17.776 elapsed=2.064s mgasps=8.612 snapdiffs=6.66MiB triediffs=181.63MiB triedirty=53.87MiB
INFO [11-16|13:37:27.366] Chain head was updated number=18,584,790 hash=0031ef..25e829 root=1ea204..1b2f2a elapsed=3.231957ms
DEBUG[11-16|13:37:36.252] Created new diff layer id=404,457 block=18,584,791 nodes=3202 size=1.21MiB
DEBUG[11-16|13:37:38.115] Stored state history id=404,329 block=18,584,663 data=21.66KiB index=32.67KiB elapsed=1.862s
DEBUG[11-16|13:37:38.117] Pruned state history items=1 tailid=314,330
INFO [11-16|13:37:38.117] Imported new potential chain segment number=18,584,791 hash=1b2685..d4a60b blocks=1 txs=114 mgas=11.122 elapsed=1.918s mgasps=5.797 snapdiffs=6.69MiB triediffs=181.16MiB triedirty=54.22MiB
INFO [11-16|13:37:38.249] Chain head was updated number=18,584,791 hash=1b2685..d4a60b root=45733a..d0af4d elapsed=2.32329ms
DEBUG[11-16|13:37:49.115] Created new diff layer id=404,458 block=18,584,792 nodes=3546 size=1.32MiB
DEBUG[11-16|13:37:51.054] Stored state history id=404,330 block=18,584,664 data=11.88KiB index=20.72KiB elapsed=1.939s
DEBUG[11-16|13:37:51.056] Pruned state history items=1 tailid=314,331
INFO [11-16|13:37:51.056] Imported new potential chain segment number=18,584,792 hash=e2e425..76564f blocks=1 txs=146 mgas=14.953 elapsed=2.013s mgasps=7.428 snapdiffs=6.73MiB triediffs=181.27MiB triedirty=54.53MiB
INFO [11-16|13:37:51.204] Chain head was updated number=18,584,792 hash=e2e425..76564f root=5cf6c7..488b32 elapsed=2.788485ms
DEBUG[11-16|13:38:00.574] Created new diff layer id=404,459 block=18,584,793 nodes=2992 size=1.13MiB
DEBUG[11-16|13:38:02.445] Stored state history id=404,331 block=18,584,665 data=13.04KiB index=21.52KiB elapsed=1.870s
DEBUG[11-16|13:38:02.447] Pruned state history items=1 tailid=314,332
INFO [11-16|13:38:02.447] Imported new potential chain segment number=18,584,793 hash=78cd51..f5b635 blocks=1 txs=137 mgas=11.700 elapsed=1.985s mgasps=5.894 snapdiffs=6.76MiB triediffs=181.18MiB triedirty=54.82MiB
INFO [11-16|13:38:02.588] Chain head was updated number=18,584,793 hash=78cd51..f5b635 root=5e0301..3189fe elapsed=2.454461ms
DEBUG[11-16|13:38:13.395] Created new diff layer id=404,460 block=18,584,794 nodes=5179 size=1.93MiB
DEBUG[11-16|13:38:13.517] Stored state history id=404,332 block=18,584,666 data=18.78KiB index=28.22KiB elapsed=121.606ms
DEBUG[11-16|13:38:13.519] Pruned state history items=1 tailid=314,333
INFO [11-16|13:38:13.519] Imported new potential chain segment number=18,584,794 hash=b699b8..e65343 blocks=1 txs=252 mgas=22.640 elapsed=197.811ms mgasps=114.450 snapdiffs=6.81MiB triediffs=181.69MiB triedirty=55.06MiB
INFO [11-16|13:38:13.656] Chain head was updated number=18,584,794 hash=b699b8..e65343 root=7d8399..4756e0 elapsed=7.415048ms
DEBUG[11-16|13:38:24.792] Created new diff layer id=404,461 block=18,584,795 nodes=3057 size=1.14MiB
DEBUG[11-16|13:38:24.915] Stored state history id=404,333 block=18,584,667 data=16.91KiB index=24.45KiB elapsed=122.811ms
DEBUG[11-16|13:38:24.917] Pruned state history items=1 tailid=314,334
INFO [11-16|13:38:24.917] Imported new potential chain segment number=18,584,795 hash=985831..7cb0ca blocks=1 txs=121 mgas=13.028 elapsed=174.045ms mgasps=74.851 snapdiffs=6.85MiB triediffs=181.51MiB triedirty=55.33MiB
INFO [11-16|13:38:25.043] Chain head was updated number=18,584,795 hash=985831..7cb0ca root=79b137..2abe16 elapsed=2.420415ms
DEBUG[11-16|13:38:38.476] Created new diff layer id=404,462 block=18,584,796 nodes=4071 size=1.51MiB
DEBUG[11-16|13:38:38.499] Stored state history id=404,334 block=18,584,668 data=13.35KiB index=22.32KiB elapsed=23.271ms
DEBUG[11-16|13:38:38.501] Pruned state history items=1 tailid=314,335
INFO [11-16|13:38:38.501] Imported new potential chain segment number=18,584,796 hash=7274fe..58a931 blocks=1 txs=159 mgas=15.252 elapsed=137.995ms mgasps=110.523 snapdiffs=6.89MiB triediffs=181.85MiB triedirty=55.63MiB
INFO [11-16|13:38:38.614] Chain head was updated number=18,584,796 hash=7274fe..58a931 root=767dcc..54454e elapsed=2.463737ms
DEBUG[11-16|13:38:48.518] Created new diff layer id=404,463 block=18,584,797 nodes=3048 size=1.14MiB
DEBUG[11-16|13:38:48.620] Stored state history id=404,335 block=18,584,669 data=18.42KiB index=31.25KiB elapsed=101.922ms
DEBUG[11-16|13:38:48.626] Pruned state history items=1 tailid=314,336
INFO [11-16|13:38:48.626] Imported new potential chain segment number=18,584,797 hash=177a24..dac144 blocks=1 txs=124 mgas=11.232 elapsed=157.408ms mgasps=71.354 snapdiffs=6.92MiB triediffs=181.27MiB triedirty=56.01MiB
INFO [11-16|13:38:48.780] Chain head was updated number=18,584,797 hash=177a24..dac144 root=3bdfd3..184781 elapsed=1.977301ms
DEBUG[11-16|13:39:01.195] Created new diff layer id=404,464 block=18,584,798 nodes=3458 size=1.29MiB
DEBUG[11-16|13:39:01.853] Stored state history id=404,336 block=18,584,670 data=11.54KiB index=16.63KiB elapsed=658.244ms
DEBUG[11-16|13:39:01.856] Pruned state history items=1 tailid=314,337
INFO [11-16|13:39:01.856] Imported new potential chain segment number=18,584,798 hash=7bcd74..5396d0 blocks=1 txs=144 mgas=15.705 elapsed=777.984ms mgasps=20.187 snapdiffs=6.96MiB triediffs=181.62MiB triedirty=56.19MiB
INFO [11-16|13:39:02.020] Chain head was updated number=18,584,798 hash=7bcd74..5396d0 root=a51a82..47f495 elapsed=3.874843ms
DEBUG[11-16|13:39:12.606] Created new diff layer id=404,465 block=18,584,799 nodes=3688 size=1.40MiB
DEBUG[11-16|13:39:14.568] Stored state history id=404,337 block=18,584,671 data=18.21KiB index=28.03KiB elapsed=1.961s
DEBUG[11-16|13:39:14.570] Pruned state history items=1 tailid=314,338
INFO [11-16|13:39:14.570] Imported new potential chain segment number=18,584,799 hash=ff3533..d375fd blocks=1 txs=172 mgas=17.670 elapsed=2.016s mgasps=8.762 snapdiffs=6.99MiB triediffs=181.53MiB triedirty=56.55MiB
INFO [11-16|13:39:14.756] Chain head was updated number=18,584,799 hash=ff3533..d375fd root=71377b..0324c8 elapsed=2.252658ms
DEBUG[11-16|13:39:25.154] Created new diff layer id=404,466 block=18,584,800 nodes=3624 size=1.37MiB
DEBUG[11-16|13:39:27.076] Stored state history id=404,338 block=18,584,672 data=21.13KiB index=31.22KiB elapsed=1.921s
DEBUG[11-16|13:39:27.078] Pruned state history items=1 tailid=314,339
INFO [11-16|13:39:27.078] Imported new potential chain segment number=18,584,800 hash=6b3a11..32a19f blocks=1 txs=161 mgas=13.011 elapsed=1.989s mgasps=6.538 snapdiffs=7.03MiB triediffs=181.28MiB triedirty=56.93MiB
INFO [11-16|13:39:27.220] Chain head was updated number=18,584,800 hash=6b3a11..32a19f root=dd50c3..c326e4 elapsed=2.488369ms
DEBUG[11-16|13:39:36.782] Created new diff layer id=404,467 block=18,584,801 nodes=3527 size=1.29MiB
DEBUG[11-16|13:39:38.645] Stored state history id=404,339 block=18,584,673 data=17.44KiB index=26.23KiB elapsed=1.863s
DEBUG[11-16|13:39:38.648] Pruned state history items=1 tailid=314,340
INFO [11-16|13:39:38.648] Imported new potential chain segment number=18,584,801 hash=29288d..0f4f20 blocks=1 txs=125 mgas=14.013 elapsed=1.926s mgasps=7.276 snapdiffs=7.07MiB triediffs=181.17MiB triedirty=57.24MiB
INFO [11-16|13:39:38.783] Chain head was updated number=18,584,801 hash=29288d..0f4f20 root=3c5eea..e01728 elapsed=3.221799ms
DEBUG[11-16|13:39:47.719] Created new diff layer id=404,468 block=18,584,802 nodes=1983 size=780.28KiB
DEBUG[11-16|13:39:49.615] Stored state history id=404,340 block=18,584,674 data=17.52KiB index=27.86KiB elapsed=1.895s
DEBUG[11-16|13:39:49.617] Pruned state history items=1 tailid=314,341
INFO [11-16|13:39:49.617] Imported new potential chain segment number=18,584,802 hash=2cc82c..fcb7c4 blocks=1 txs=90 mgas=6.996 elapsed=1.923s mgasps=3.637 snapdiffs=7.09MiB triediffs=180.40MiB triedirty=57.59MiB
INFO [11-16|13:39:49.764] Chain head was updated number=18,584,802 hash=2cc82c..fcb7c4 root=bfe91b..ebac74 elapsed=1.959168ms
DEBUG[11-16|13:40:00.972] Created new diff layer id=404,469 block=18,584,803 nodes=4740 size=1.75MiB
DEBUG[11-16|13:40:03.021] Stored state history id=404,341 block=18,584,675 data=15.82KiB index=25.14KiB elapsed=2.049s
DEBUG[11-16|13:40:03.023] Pruned state history items=1 tailid=314,342
INFO [11-16|13:40:03.024] Imported new potential chain segment number=18,584,803 hash=247d4a..cd60b7 blocks=1 txs=203 mgas=28.047 elapsed=2.276s mgasps=12.320 snapdiffs=7.14MiB triediffs=180.77MiB triedirty=57.88MiB
INFO [11-16|13:40:03.239] Chain head was updated number=18,584,803 hash=247d4a..cd60b7 root=a74c50..74dfab elapsed=6.935127ms
DEBUG[11-16|13:40:12.326] Created new diff layer id=404,470 block=18,584,804 nodes=3446 size=1.30MiB
DEBUG[11-16|13:40:12.442] Stored state history id=404,342 block=18,584,676 data=13.99KiB index=21.66KiB elapsed=115.258ms
DEBUG[11-16|13:40:12.444] Pruned state history items=1 tailid=314,343
INFO [11-16|13:40:12.444] Imported new potential chain segment number=18,584,804 hash=3d75f1..e7aa9f blocks=1 txs=144 mgas=19.935 elapsed=209.285ms mgasps=95.253 snapdiffs=7.18MiB triediffs=180.86MiB triedirty=58.17MiB
INFO [11-16|13:40:12.577] Chain head was updated number=18,584,804 hash=3d75f1..e7aa9f root=e5fccf..ac2eee elapsed=3.354355ms
DEBUG[11-16|13:40:24.616] Created new diff layer id=404,471 block=18,584,805 nodes=2721 size=1.01MiB
DEBUG[11-16|13:40:24.734] Stored state history id=404,343 block=18,584,677 data=19.35KiB index=29.42KiB elapsed=117.859ms
DEBUG[11-16|13:40:24.736] Pruned state history items=1 tailid=314,344
INFO [11-16|13:40:24.737] Imported new potential chain segment number=18,584,805 hash=52bc82..56c9aa blocks=1 txs=118 mgas=14.053 elapsed=180.825ms mgasps=77.718 snapdiffs=7.21MiB triediffs=180.28MiB triedirty=58.48MiB
INFO [11-16|13:40:24.857] Chain head was updated number=18,584,805 hash=52bc82..56c9aa root=313816..16d6b6 elapsed=2.837733ms
DEBUG[11-16|13:40:36.351] Created new diff layer id=404,472 block=18,584,806 nodes=3048 size=1.15MiB
DEBUG[11-16|13:40:36.409] Stored state history id=404,344 block=18,584,678 data=14.73KiB index=22.84KiB elapsed=58.184ms
DEBUG[11-16|13:40:36.411] Pruned state history items=1 tailid=314,345
INFO [11-16|13:40:36.411] Imported new potential chain segment number=18,584,806 hash=e87e5d..2db714 blocks=1 txs=104 mgas=10.702 elapsed=123.281ms mgasps=86.808 snapdiffs=7.24MiB triediffs=180.19MiB triedirty=58.77MiB
INFO [11-16|13:40:36.528] Chain head was updated number=18,584,806 hash=e87e5d..2db714 root=ab96ae..1b3c1d elapsed=1.903847ms
DEBUG[11-16|13:40:48.420] Created new diff layer id=404,473 block=18,584,807 nodes=4212 size=1.55MiB
DEBUG[11-16|13:40:48.444] Stored state history id=404,345 block=18,584,679 data=20.17KiB index=33.63KiB elapsed=23.531ms
DEBUG[11-16|13:40:48.446] Pruned state history items=1 tailid=314,346
INFO [11-16|13:40:48.446] Imported new potential chain segment number=18,584,807 hash=1a342b..3a18e4 blocks=1 txs=174 mgas=17.958 elapsed=131.247ms mgasps=136.829 snapdiffs=7.28MiB triediffs=179.84MiB triedirty=59.15MiB
INFO [11-16|13:40:48.590] Chain head was updated number=18,584,807 hash=1a342b..3a18e4 root=42ad15..463d67 elapsed=2.356568ms
DEBUG[11-16|13:41:00.143] Created new diff layer id=404,474 block=18,584,808 nodes=2741 size=1.04MiB
DEBUG[11-16|13:41:00.436] Stored state history id=404,346 block=18,584,680 data=21.19KiB index=31.63KiB elapsed=292.867ms
DEBUG[11-16|13:41:00.439] Pruned state history items=1 tailid=314,347
INFO [11-16|13:41:00.439] Imported new potential chain segment number=18,584,808 hash=b5190b..a30451 blocks=1 txs=118 mgas=18.210 elapsed=334.501ms mgasps=54.440 snapdiffs=7.31MiB triediffs=179.18MiB triedirty=59.48MiB
INFO [11-16|13:41:00.555] Chain head was updated number=18,584,808 hash=b5190b..a30451 root=fbcf66..b74654 elapsed=3.849347ms
DEBUG[11-16|13:41:12.787] Created new diff layer id=404,475 block=18,584,809 nodes=4234 size=1.59MiB
DEBUG[11-16|13:41:14.752] Stored state history id=404,347 block=18,584,681 data=16.61KiB index=24.77KiB elapsed=1.964s
DEBUG[11-16|13:41:14.754] Pruned state history items=1 tailid=314,348
INFO [11-16|13:41:14.754] Imported new potential chain segment number=18,584,809 hash=aaa293..841008 blocks=1 txs=153 mgas=18.150 elapsed=2.093s mgasps=8.668 snapdiffs=7.35MiB triediffs=179.39MiB triedirty=59.73MiB
INFO [11-16|13:41:14.870] Chain head was updated number=18,584,809 hash=aaa293..841008 root=13cfc3..74d6c3 elapsed=3.566993ms
DEBUG[11-16|13:41:24.128] Created new diff layer id=404,476 block=18,584,810 nodes=3169 size=1.19MiB
DEBUG[11-16|13:41:26.087] Stored state history id=404,348 block=18,584,682 data=20.28KiB index=29.73KiB elapsed=1.958s
DEBUG[11-16|13:41:26.100] Pruned state history items=1 tailid=314,349
INFO [11-16|13:41:26.100] Imported new potential chain segment number=18,584,810 hash=bd2e89..63e534 blocks=1 txs=128 mgas=11.594 elapsed=2.026s mgasps=5.721 snapdiffs=7.39MiB triediffs=179.04MiB triedirty=60.02MiB
INFO [11-16|13:41:26.211] Chain head was updated number=18,584,810 hash=bd2e89..63e534 root=4bf938..2a10ec elapsed=2.528813ms
DEBUG[11-16|13:41:36.566] Created new diff layer id=404,477 block=18,584,811 nodes=2454 size=948.14KiB
DEBUG[11-16|13:41:38.438] Stored state history id=404,349 block=18,584,683 data=26.77KiB index=39.21KiB elapsed=1.871s
DEBUG[11-16|13:41:38.440] Pruned state history items=1 tailid=314,350
INFO [11-16|13:41:38.440] Imported new potential chain segment number=18,584,811 hash=18db0d..907db1 blocks=1 txs=103 mgas=8.433 elapsed=1.924s mgasps=4.381 snapdiffs=7.41MiB triediffs=177.96MiB triedirty=60.44MiB
INFO [11-16|13:41:38.552] Chain head was updated number=18,584,811 hash=18db0d..907db1 root=9b9a0d..ce0f99 elapsed=2.252336ms
DEBUG[11-16|13:41:48.797] Created new diff layer id=404,478 block=18,584,812 nodes=3471 size=1.29MiB
DEBUG[11-16|13:41:50.693] Stored state history id=404,350 block=18,584,684 data=21.01KiB index=32.27KiB elapsed=1.896s
DEBUG[11-16|13:41:50.695] Pruned state history items=1 tailid=314,351
INFO [11-16|13:41:50.695] Imported new potential chain segment number=18,584,812 hash=97db16..e8b284 blocks=1 txs=142 mgas=14.622 elapsed=1.948s mgasps=7.503 snapdiffs=7.45MiB triediffs=177.52MiB triedirty=60.86MiB
INFO [11-16|13:41:50.859] Chain head was updated number=18,584,812 hash=97db16..e8b284 root=7400c1..fae59b elapsed=4.319908ms
DEBUG[11-16|13:42:01.474] Created new diff layer id=404,479 block=18,584,813 nodes=3333 size=1.24MiB
DEBUG[11-16|13:42:03.516] Stored state history id=404,351 block=18,584,685 data=26.72KiB index=40.19KiB elapsed=2.041s
DEBUG[11-16|13:42:03.519] Pruned state history items=1 tailid=314,352
INFO [11-16|13:42:03.519] Imported new potential chain segment number=18,584,813 hash=15f017..502a5e blocks=1 txs=128 mgas=11.198 elapsed=2.112s mgasps=5.300 snapdiffs=7.48MiB triediffs=176.65MiB triedirty=61.32MiB
INFO [11-16|13:42:03.643] Chain head was updated number=18,584,813 hash=15f017..502a5e root=cfe03e..4341da elapsed=5.300922ms
DEBUG[11-16|13:42:13.999] Created new diff layer id=404,480 block=18,584,814 nodes=4389 size=1.64MiB
DEBUG[11-16|13:42:14.123] Stored state history id=404,352 block=18,584,686 data=29.06KiB index=41.36KiB elapsed=124.654ms
DEBUG[11-16|13:42:14.126] Pruned state history items=1 tailid=314,353
INFO [11-16|13:42:14.126] Imported new potential chain segment number=18,584,814 hash=d048eb..d366d7 blocks=1 txs=170 mgas=16.575 elapsed=211.029ms mgasps=78.543 snapdiffs=7.53MiB triediffs=176.18MiB triedirty=61.76MiB
INFO [11-16|13:42:14.220] Chain head was updated number=18,584,814 hash=d048eb..d366d7 root=0d843b..6a834f elapsed=2.779978ms
DEBUG[11-16|13:42:24.126] Created new diff layer id=404,481 block=18,584,815 nodes=3304 size=1.24MiB
DEBUG[11-16|13:42:24.244] Stored state history id=404,353 block=18,584,687 data=22.04KiB index=32.76KiB elapsed=118.181ms
DEBUG[11-16|13:42:24.247] Pruned state history items=1 tailid=314,354
INFO [11-16|13:42:24.247] Imported new potential chain segment number=18,584,815 hash=dea5bb..22b407 blocks=1 txs=141 mgas=13.158 elapsed=185.646ms mgasps=70.877 snapdiffs=7.56MiB triediffs=175.77MiB triedirty=62.11MiB
INFO [11-16|13:42:24.345] Chain head was updated number=18,584,815 hash=dea5bb..22b407 root=1b3e4e..6fa147 elapsed=2.164702ms
DEBUG[11-16|13:42:35.539] Created new diff layer id=404,482 block=18,584,816 nodes=2714 size=1.03MiB
DEBUG[11-16|13:42:35.580] Stored state history id=404,354 block=18,584,688 data=15.95KiB index=23.64KiB elapsed=40.981ms
DEBUG[11-16|13:42:35.582] Pruned state history items=1 tailid=314,355
INFO [11-16|13:42:35.582] Imported new potential chain segment number=18,584,816 hash=3a243a..3dd21e blocks=1 txs=114 mgas=9.838 elapsed=79.142ms mgasps=124.311 snapdiffs=7.59MiB triediffs=175.58MiB triedirty=62.35MiB
INFO [11-16|13:42:35.693] Chain head was updated number=18,584,816 hash=3a243a..3dd21e root=92a191..1ddcfc elapsed=9.223231ms
DEBUG[11-16|13:42:50.021] Created new diff layer id=404,483 block=18,584,817 nodes=4011 size=1.50MiB
DEBUG[11-16|13:42:50.101] Stored state history id=404,355 block=18,584,689 data=16.45KiB index=25.12KiB elapsed=80.199ms
DEBUG[11-16|13:42:50.103] Pruned state history items=1 tailid=314,356
INFO [11-16|13:42:50.103] Imported new potential chain segment number=18,584,817 hash=0b539e..c88229 blocks=1 txs=178 mgas=27.036 elapsed=208.274ms mgasps=129.810 snapdiffs=8.07MiB triediffs=175.67MiB triedirty=62.66MiB
INFO [11-16|13:42:51.656] Chain head was updated number=18,584,817 hash=0b539e..c88229 root=5c75f7..3d8871 elapsed=4.758718ms
DEBUG[11-16|13:43:01.000] Created new diff layer id=404,484 block=18,584,818 nodes=3263 size=1.21MiB
DEBUG[11-16|13:43:01.714] Stored state history id=404,356 block=18,584,690 data=17.38KiB index=26.86KiB elapsed=714.099ms
DEBUG[11-16|13:43:01.716] Pruned state history items=1 tailid=314,357
INFO [11-16|13:43:01.716] Imported new potential chain segment number=18,584,818 hash=4322ae..60829a blocks=1 txs=138 mgas=13.124 elapsed=786.451ms mgasps=16.688 snapdiffs=8.10MiB triediffs=175.46MiB triedirty=62.96MiB
INFO [11-16|13:43:01.883] Chain head was updated number=18,584,818 hash=4322ae..60829a root=4c42b2..654e06 elapsed=2.357591ms
DEBUG[11-16|13:43:12.684] Created new diff layer id=404,485 block=18,584,819 nodes=3479 size=1.32MiB
DEBUG[11-16|13:43:14.644] Stored state history id=404,357 block=18,584,691 data=14.62KiB index=21.71KiB elapsed=1.959s
DEBUG[11-16|13:43:14.646] Pruned state history items=1 tailid=314,358
INFO [11-16|13:43:14.647] Imported new potential chain segment number=18,584,819 hash=201489..b1d4c6 blocks=1 txs=154 mgas=13.120 elapsed=2.023s mgasps=6.484 snapdiffs=8.14MiB triediffs=175.55MiB triedirty=63.21MiB
INFO [11-16|13:43:14.845] Chain head was updated number=18,584,819 hash=201489..b1d4c6 root=51abc2..5d3c3a elapsed=2.801163ms
DEBUG[11-16|13:43:25.618] Created new diff layer id=404,486 block=18,584,820 nodes=2696 size=1.02MiB
DEBUG[11-16|13:43:27.571] Stored state history id=404,358 block=18,584,692 data=16.83KiB index=25.00KiB elapsed=1.953s
DEBUG[11-16|13:43:27.573] Pruned state history items=1 tailid=314,359
INFO [11-16|13:43:27.573] Imported new potential chain segment number=18,584,820 hash=6ab980..f137b1 blocks=1 txs=117 mgas=15.579 elapsed=2.003s mgasps=7.776 snapdiffs=8.16MiB triediffs=175.22MiB triedirty=63.44MiB
INFO [11-16|13:43:27.726] Chain head was updated number=18,584,820 hash=6ab980..f137b1 root=131b04..2ee157 elapsed=4.587175ms
DEBUG[11-16|13:43:36.343] Created new diff layer id=404,487 block=18,584,821 nodes=3201 size=1.21MiB
DEBUG[11-16|13:43:38.212] Stored state history id=404,359 block=18,584,693 data=12.97KiB index=19.09KiB elapsed=1.868s
DEBUG[11-16|13:43:38.214] Pruned state history items=1 tailid=314,360
INFO [11-16|13:43:38.214] Imported new potential chain segment number=18,584,821 hash=aaa2d6..a7ae0a blocks=1 txs=132 mgas=10.886 elapsed=1.930s mgasps=5.640 snapdiffs=8.20MiB triediffs=175.33MiB triedirty=63.68MiB
INFO [11-16|13:43:38.348] Chain head was updated number=18,584,821 hash=aaa2d6..a7ae0a root=781806..415029 elapsed=2.58059ms
DEBUG[11-16|13:43:48.658] Created new diff layer id=404,488 block=18,584,822 nodes=3266 size=1.24MiB
DEBUG[11-16|13:43:50.540] Stored state history id=404,360 block=18,584,694 data=23.97KiB index=40.36KiB elapsed=1.882s
DEBUG[11-16|13:43:51.006] Persisted pathdb nodes nodes=3615 bytes=68.35MiB elapsed=463.936ms
DEBUG[11-16|13:43:51.007] Pruned state history items=1 tailid=314,361
INFO [11-16|13:43:51.007] Imported new potential chain segment number=18,584,822 hash=536f38..ca16be blocks=1 txs=126 mgas=12.399 elapsed=2.403s mgasps=5.159 snapdiffs=8.23MiB triediffs=174.44MiB triedirty=0.00B
INFO [11-16|13:43:51.175] Chain head was updated number=18,584,822 hash=536f38..ca16be root=57a112..c235b3 elapsed=3.611366ms
DEBUG[11-16|13:44:02.057] Created new diff layer id=404,489 block=18,584,823 nodes=3428 size=1.28MiB
DEBUG[11-16|13:44:04.112] Stored state history id=404,361 block=18,584,695 data=13.34KiB index=19.55KiB elapsed=2.054s
DEBUG[11-16|13:44:04.114] Pruned state history items=1 tailid=314,362
INFO [11-16|13:44:04.114] Imported new potential chain segment number=18,584,823 hash=e4f6b1..29db53 blocks=1 txs=148 mgas=12.605 elapsed=2.105s mgasps=5.988 snapdiffs=8.27MiB triediffs=174.67MiB triedirty=961.22KiB
INFO [11-16|13:44:05.036] Chain head was updated number=18,584,823 hash=e4f6b1..29db53 root=4b665b..15fa40 elapsed=1.897812ms
DEBUG[11-16|13:44:14.777] Created new diff layer id=404,490 block=18,584,824 nodes=4155 size=1.55MiB
DEBUG[11-16|13:44:14.894] Stored state history id=404,362 block=18,584,696 data=17.80KiB index=27.33KiB elapsed=116.577ms
DEBUG[11-16|13:44:14.895] Pruned state history items=1 tailid=314,363
INFO [11-16|13:44:14.895] Imported new potential chain segment number=18,584,824 hash=042c6d..120cc2 blocks=1 txs=171 mgas=27.457 elapsed=304.155ms mgasps=90.273 snapdiffs=8.31MiB triediffs=174.76MiB triedirty=2.00MiB
INFO [11-16|13:44:15.047] Chain head was updated number=18,584,824 hash=042c6d..120cc2 root=ed0535..7abc34 elapsed=5.084482ms
DEBUG[11-16|13:44:24.521] Created new diff layer id=404,491 block=18,584,825 nodes=2884 size=1.07MiB
DEBUG[11-16|13:44:24.642] Stored state history id=404,363 block=18,584,697 data=14.07KiB index=21.79KiB elapsed=120.796ms
DEBUG[11-16|13:44:24.644] Pruned state history items=1 tailid=314,364
INFO [11-16|13:44:24.644] Imported new potential chain segment number=18,584,825 hash=546bf4..d9f25f blocks=1 txs=116 mgas=10.390 elapsed=161.791ms mgasps=64.218 snapdiffs=8.34MiB triediffs=174.58MiB triedirty=2.80MiB
INFO [11-16|13:44:24.764] Chain head was updated number=18,584,825 hash=546bf4..d9f25f root=23925e..79ca5e elapsed=1.838844ms
DEBUG[11-16|13:44:36.344] Created new diff layer id=404,492 block=18,584,826 nodes=3222 size=1.21MiB
DEBUG[11-16|13:44:36.407] Stored state history id=404,364 block=18,584,698 data=15.58KiB index=24.39KiB elapsed=62.954ms
DEBUG[11-16|13:44:36.408] Pruned state history items=1 tailid=314,365
INFO [11-16|13:44:36.408] Imported new potential chain segment number=18,584,826 hash=346095..4385e5 blocks=1 txs=115 mgas=12.632 elapsed=113.988ms mgasps=110.820 snapdiffs=8.38MiB triediffs=174.43MiB triedirty=3.61MiB
INFO [11-16|13:44:36.563] Chain head was updated number=18,584,826 hash=346095..4385e5 root=0b91ce..82d6b8 elapsed=2.763176ms
DEBUG[11-16|13:44:48.980] Created new diff layer id=404,493 block=18,584,827 nodes=3386 size=1.28MiB
DEBUG[11-16|13:44:49.003] Stored state history id=404,365 block=18,584,699 data=18.13KiB index=28.74KiB elapsed=23.171ms
DEBUG[11-16|13:44:49.005] Pruned state history items=1 tailid=314,366
INFO [11-16|13:44:49.005] Imported new potential chain segment number=18,584,827 hash=4ac484..4cfec8 blocks=1 txs=144 mgas=11.609 elapsed=105.565ms mgasps=109.965 snapdiffs=8.41MiB triediffs=174.15MiB triedirty=4.45MiB
INFO [11-16|13:44:49.144] Chain head was updated number=18,584,827 hash=4ac484..4cfec8 root=959908..60c76a elapsed=1.980118ms
DEBUG[11-16|13:45:01.192] Created new diff layer id=404,494 block=18,584,828 nodes=3827 size=1.42MiB
DEBUG[11-16|13:45:01.321] Stored state history id=404,366 block=18,584,700 data=13.22KiB index=19.69KiB elapsed=129.501ms
DEBUG[11-16|13:45:01.324] Pruned state history items=1 tailid=314,367
INFO [11-16|13:45:01.324] Imported new potential chain segment number=18,584,828 hash=b471a0..13495b blocks=1 txs=155 mgas=26.900 elapsed=255.881ms mgasps=105.125 snapdiffs=8.45MiB triediffs=174.50MiB triedirty=5.00MiB
INFO [11-16|13:45:01.474] Chain head was updated number=18,584,828 hash=b471a0..13495b root=3268a1..4f44c7 elapsed=4.471381ms
DEBUG[11-16|13:45:12.372] Created new diff layer id=404,495 block=18,584,829 nodes=3148 size=1.20MiB
DEBUG[11-16|13:45:14.323] Stored state history id=404,367 block=18,584,701 data=15.10KiB index=24.47KiB elapsed=1.950s
DEBUG[11-16|13:45:14.325] Pruned state history items=1 tailid=314,368
INFO [11-16|13:45:14.325] Imported new potential chain segment number=18,584,829 hash=8e1641..43f6a4 blocks=1 txs=137 mgas=10.571 elapsed=2.001s mgasps=5.281 snapdiffs=8.48MiB triediffs=174.32MiB triedirty=5.75MiB
INFO [11-16|13:45:14.466] Chain head was updated number=18,584,829 hash=8e1641..43f6a4 root=5c871d..eebb66 elapsed=2.765169ms
DEBUG[11-16|13:45:24.532] Created new diff layer id=404,496 block=18,584,830 nodes=4206 size=1.55MiB
DEBUG[11-16|13:45:26.499] Stored state history id=404,368 block=18,584,702 data=14.70KiB index=21.48KiB elapsed=1.966s
DEBUG[11-16|13:45:26.501] Pruned state history items=1 tailid=314,369
INFO [11-16|13:45:26.501] Imported new potential chain segment number=18,584,830 hash=aeb84e..d4730a blocks=1 txs=160 mgas=16.380 elapsed=2.058s mgasps=7.956 snapdiffs=8.53MiB triediffs=174.72MiB triedirty=6.35MiB
INFO [11-16|13:45:26.664] Chain head was updated number=18,584,830 hash=aeb84e..d4730a root=5dd960..117a0c elapsed=3.535392ms
DEBUG[11-16|13:45:36.670] Created new diff layer id=404,497 block=18,584,831 nodes=3247 size=1.21MiB
DEBUG[11-16|13:45:38.576] Stored state history id=404,369 block=18,584,703 data=15.74KiB index=23.56KiB elapsed=1.905s
DEBUG[11-16|13:45:38.577] Pruned state history items=1 tailid=314,370
INFO [11-16|13:45:38.577] Imported new potential chain segment number=18,584,831 hash=c2c7fd..4f9370 blocks=1 txs=125 mgas=11.643 elapsed=1.963s mgasps=5.931 snapdiffs=8.56MiB triediffs=174.64MiB triedirty=6.93MiB
INFO [11-16|13:45:38.714] Chain head was updated number=18,584,831 hash=c2c7fd..4f9370 root=b91172..f2103d elapsed=2.692548ms
DEBUG[11-16|13:45:48.873] Created new diff layer id=404,498 block=18,584,832 nodes=3860 size=1.47MiB
DEBUG[11-16|13:45:50.738] Stored state history id=404,370 block=18,584,704 data=16.27KiB index=28.43KiB elapsed=1.864s
DEBUG[11-16|13:45:50.740] Pruned state history items=1 tailid=314,371
INFO [11-16|13:45:50.740] Imported new potential chain segment number=18,584,832 hash=4096b0..938e65 blocks=1 txs=202 mgas=12.937 elapsed=1.931s mgasps=6.698 snapdiffs=8.60MiB triediffs=174.46MiB triedirty=7.80MiB
INFO [11-16|13:45:50.869] Chain head was updated number=18,584,832 hash=4096b0..938e65 root=8915ad..8ca209 elapsed=2.270656ms
DEBUG[11-16|13:46:01.313] Created new diff layer id=404,499 block=18,584,833 nodes=3435 size=1.26MiB
DEBUG[11-16|13:46:03.285] Stored state history id=404,371 block=18,584,705 data=16.93KiB index=25.45KiB elapsed=1.970s
DEBUG[11-16|13:46:03.292] Pruned state history items=1 tailid=314,372
INFO [11-16|13:46:03.292] Imported new potential chain segment number=18,584,833 hash=488a24..892218 blocks=1 txs=121 mgas=15.862 elapsed=2.150s mgasps=7.375 snapdiffs=8.64MiB triediffs=174.31MiB triedirty=8.39MiB
INFO [11-16|13:46:03.451] Chain head was updated number=18,584,833 hash=488a24..892218 root=b9bca8..781a37 elapsed=2.872236ms
DEBUG[11-16|13:46:12.647] Created new diff layer id=404,500 block=18,584,834 nodes=3899 size=1.47MiB
DEBUG[11-16|13:46:13.336] Stored state history id=404,372 block=18,584,706 data=14.84KiB index=22.27KiB elapsed=689.517ms
DEBUG[11-16|13:46:13.338] Pruned state history items=1 tailid=314,373
INFO [11-16|13:46:13.338] Imported new potential chain segment number=18,584,834 hash=98fa77..6e97fe blocks=1 txs=166 mgas=14.568 elapsed=755.546ms mgasps=19.282 snapdiffs=8.68MiB triediffs=174.52MiB triedirty=8.95MiB
INFO [11-16|13:46:13.497] Chain head was updated number=18,584,834 hash=98fa77..6e97fe root=4f32ee..af1530 elapsed=2.656991ms
DEBUG[11-16|13:46:24.346] Created new diff layer id=404,501 block=18,584,835 nodes=2958 size=1.12MiB
DEBUG[11-16|13:46:24.466] Stored state history id=404,373 block=18,584,707 data=17.77KiB index=28.55KiB elapsed=119.858ms
DEBUG[11-16|13:46:24.468] Pruned state history items=1 tailid=314,374
INFO [11-16|13:46:24.468] Imported new potential chain segment number=18,584,835 hash=155f2f..97641e blocks=1 txs=121 mgas=10.599 elapsed=188.020ms mgasps=56.372 snapdiffs=8.71MiB triediffs=174.09MiB triedirty=9.63MiB
INFO [11-16|13:46:24.594] Chain head was updated number=18,584,835 hash=155f2f..97641e root=4e900f..d97edc elapsed=1.906175ms
DEBUG[11-16|13:46:38.396] Created new diff layer id=404,502 block=18,584,836 nodes=4352 size=1.58MiB
DEBUG[11-16|13:46:38.475] Stored state history id=404,374 block=18,584,708 data=20.26KiB index=34.18KiB elapsed=78.537ms
DEBUG[11-16|13:46:38.477] Pruned state history items=1 tailid=314,375
INFO [11-16|13:46:38.477] Imported new potential chain segment number=18,584,836 hash=4ee278..5344e1 blocks=1 txs=161 mgas=25.479 elapsed=196.926ms mgasps=129.381 snapdiffs=8.76MiB triediffs=173.68MiB triedirty=10.75MiB
INFO [11-16|13:46:38.620] Chain head was updated number=18,584,836 hash=4ee278..5344e1 root=99dd4e..dff40e elapsed=3.75079ms
DEBUG[11-16|13:46:48.995] Created new diff layer id=404,503 block=18,584,837 nodes=4151 size=1.54MiB
DEBUG[11-16|13:46:49.027] Stored state history id=404,375 block=18,584,709 data=17.90KiB index=27.70KiB elapsed=32.306ms
DEBUG[11-16|13:46:49.029] Pruned state history items=1 tailid=314,376
INFO [11-16|13:46:49.029] Imported new potential chain segment number=18,584,837 hash=16ab61..4bdeee blocks=1 txs=144 mgas=16.614 elapsed=119.885ms mgasps=138.579 snapdiffs=8.80MiB triediffs=173.67MiB triedirty=11.42MiB
INFO [11-16|13:46:49.146] Chain head was updated number=18,584,837 hash=16ab61..4bdeee root=c09ee1..773d17 elapsed=2.581797ms
DEBUG[11-16|13:47:02.172] Created new diff layer id=404,504 block=18,584,838 nodes=3595 size=1.35MiB
DEBUG[11-16|13:47:02.267] Stored state history id=404,376 block=18,584,710 data=14.84KiB index=22.10KiB elapsed=95.067ms
DEBUG[11-16|13:47:02.269] Pruned state history items=1 tailid=314,377
INFO [11-16|13:47:02.269] Imported new potential chain segment number=18,584,838 hash=87c902..7a30c3 blocks=1 txs=143 mgas=13.079 elapsed=166.876ms mgasps=78.373 snapdiffs=8.84MiB triediffs=173.82MiB triedirty=11.88MiB
INFO [11-16|13:47:02.386] Chain head was updated number=18,584,838 hash=87c902..7a30c3 root=ca96f2..9942d0 elapsed=2.00757ms
DEBUG[11-16|13:47:12.812] Created new diff layer id=404,505 block=18,584,839 nodes=2956 size=1.11MiB
DEBUG[11-16|13:47:14.719] Stored state history id=404,377 block=18,584,711 data=15.23KiB index=23.26KiB elapsed=1.906s
DEBUG[11-16|13:47:14.721] Pruned state history items=1 tailid=314,378
INFO [11-16|13:47:14.721] Imported new potential chain segment number=18,584,839 hash=f9ac6d..2f4562 blocks=1 txs=111 mgas=12.630 elapsed=1.985s mgasps=6.360 snapdiffs=8.87MiB triediffs=173.61MiB triedirty=12.42MiB
INFO [11-16|13:47:14.860] Chain head was updated number=18,584,839 hash=f9ac6d..2f4562 root=d91086..5a4e16 elapsed=3.391154ms
As a friendly reminder, I don't see any of these poor attestation performance issues when running geth v1.13.4.
In simple terms:
My geth data directory lives at /var/lib/geth
- 4TB WD_BLACK SN850X, mounted at
/var - 2TB Samsung SSD 860, mounted at
/var/lib/geth/geth/chaindata/ancient
Hardware specs in full:
$ sudo lshw gives:
nuc
description: Mini PC
product: NUC11PAHi7 (RNUC11PAHi70Z0)
vendor: Intel(R) Client Systems
version: M68291-500
serial: G6PA244007JL
width: 64 bits
capabilities: smbios-3.3.0 dmi-3.3.0 smp vsyscall32
configuration: boot=normal chassis=mini family=PA sku=RNUC11PAHi70Z0 uuid=c692bedc-52aa-94e3-f317-88aedd0dc9af
*-core
description: Motherboard
product: NUC11PABi7
vendor: Intel Corporation
physical id: 0
version: M68262-500
serial: GEPA243005VZ
*-firmware
description: BIOS
vendor: Intel Corp.
physical id: 0
version: PATGL357.0047.2022.0627.1120
date: 06/27/2022
size: 64KiB
capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppynec int13floppytoshiba int13floppy360 int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int14serial int17printer int10video acpi usb biosbootspecification uefi
*-memory
description: System Memory
physical id: 5b
slot: System board or motherboard
size: 32GiB
*-bank:0
description: SODIMM DDR4 Synchronous 3200 MHz (0.3 ns)
product: PSD416G320081S
vendor: Patriot Memory (PDP Systems)
physical id: 0
serial: 052601C8
slot: SODIMM1
size: 16GiB
width: 64 bits
clock: 3200MHz (0.3ns)
*-bank:1
description: SODIMM DDR4 Synchronous 3200 MHz (0.3 ns)
product: PSD416G320081S
vendor: Patriot Memory (PDP Systems)
physical id: 1
serial: 05260296
slot: SODIMM2
size: 16GiB
width: 64 bits
clock: 3200MHz (0.3ns)
*-cache:0
description: L1 cache
physical id: 6a
slot: L1 Cache
size: 192KiB
capacity: 192KiB
capabilities: synchronous internal write-back data
configuration: level=1
*-cache:1
description: L1 cache
physical id: 6b
slot: L1 Cache
size: 128KiB
capacity: 128KiB
capabilities: synchronous internal write-back instruction
configuration: level=1
*-cache:2
description: L2 cache
physical id: 6c
slot: L2 Cache
size: 5MiB
capacity: 5MiB
capabilities: synchronous internal write-back unified
configuration: level=2
*-cache:3
description: L3 cache
physical id: 6d
slot: L3 Cache
size: 12MiB
capacity: 12MiB
capabilities: synchronous internal write-back unified
configuration: level=3
*-cpu
description: CPU
product: 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
vendor: Intel Corp.
physical id: 6e
bus info: cpu@0
version: 6.140.1
serial: To Be Filled By O.E.M.
slot: U3E1
size: 3260MHz
capacity: 4700MHz
width: 64 bits
clock: 100MHz
capabilities: lm fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear flush_l1d arch_capabilities cpufreq
configuration: cores=4 enabledcores=4 microcode=172 threads=8
*-pci
description: Host bridge
product: 11th Gen Core Processor Host Bridge/DRAM Registers
vendor: Intel Corporation
physical id: 100
bus info: pci@0000:00:00.0
version: 01
width: 32 bits
clock: 33MHz
*-display
description: VGA compatible controller
product: TigerLake-LP GT2 [Iris Xe Graphics]
vendor: Intel Corporation
physical id: 2
bus info: pci@0000:00:02.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm vga_controller bus_master cap_list rom
configuration: driver=i915 latency=0
resources: iomemory:600-5ff iomemory:400-3ff irq:179 memory:603c000000-603cffffff memory:4000000000-400fffffff ioport:3000(size=64) memory:c0000-dffff memory:4010000000-4016ffffff memory:4020000000-40ffffffff
*-pci:0
description: PCI bridge
product: 11th Gen Core Processor PCIe Controller
vendor: Intel Corporation
physical id: 6
bus info: pci@0000:00:06.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:124 memory:6a600000-6a6fffff
*-nvme
description: NVMe device
product: WD_BLACK SN850X 4000GB
vendor: Sandisk Corp
physical id: 0
bus info: pci@0000:01:00.0
logical name: /dev/nvme0
version: 624311WD
serial: 22440Q801175
width: 64 bits
clock: 33MHz
capabilities: nvme pm msi msix pciexpress nvm_express bus_master cap_list
configuration: driver=nvme latency=0 nqn=nqn.2018-01.com.wdc:nguid:E8238FA6BF53-0001-001B448B4E1BFA03 state=live
resources: irq:16 memory:6a600000-6a603fff
*-namespace:0
description: NVMe disk
physical id: 0
logical name: hwmon1
*-namespace:1
description: NVMe disk
physical id: 2
logical name: /dev/ng0n1
*-namespace:2
description: NVMe disk
physical id: 1
bus info: nvme@0:1
logical name: /dev/nvme0n1
size: 3726GiB (4TB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: guid=cfa5a833-b0d2-4147-952e-7ee748092089 logicalsectorsize=512 sectorsize=512 wwid=eui.e8238fa6bf530001001b448b4e1bfa03
*-volume:0
description: Windows FAT volume
vendor: mkfs.fat
physical id: 1
bus info: nvme@0:1,1
logical name: /dev/nvme0n1p1
logical name: /boot/efi
version: FAT32
serial: b5f5-ab4e
size: 1073MiB
capacity: 1074MiB
capabilities: boot fat initialized
configuration: FATs=2 filesystem=fat mount.fstype=vfat mount.options=rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro state=mounted
*-volume:1
description: EXT4 volume
vendor: Linux
physical id: 2
bus info: nvme@0:1,2
logical name: /dev/nvme0n1p2
logical name: /boot
version: 1.0
serial: 30dfe47f-d67e-4e05-8675-c3dfd5eefbeb
size: 1GiB
capabilities: journaled extended_attributes large_files huge_files dir_nlink recover 64bit extents ext4 ext2 initialized
configuration: created=2022-12-28 23:04:01 filesystem=ext4 lastmountpoint=/boot modified=2023-11-14 15:41:32 mount.fstype=ext4 mount.options=rw,relatime mounted=2023-11-14 15:41:32 state=mounted
*-volume:2
description: EFI partition
physical id: 3
bus info: nvme@0:1,3
logical name: /dev/nvme0n1p3
serial: 27214W-VMXm-pvU5-m2N4-rVfu-u0Co-x3frLs
size: 3723GiB
capabilities: lvm2
*-pci:1
description: PCI bridge
product: Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #0
vendor: Intel Corporation
physical id: 7
bus info: pci@0000:00:07.0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:125 ioport:4000(size=4096) memory:5e000000-6a1fffff ioport:6020000000(size=469762048)
*-pci:2
description: PCI bridge
product: Tiger Lake-LP Thunderbolt 4 PCI Express Root Port #3
vendor: Intel Corporation
physical id: 7.3
bus info: pci@0000:00:07.3
version: 01
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:126 ioport:5000(size=4096) memory:50000000-5c1fffff ioport:6000000000(size=469762048)
*-generic UNCLAIMED
description: System peripheral
product: GNA Scoring Accelerator module
vendor: Intel Corporation
physical id: 8
bus info: pci@0000:00:08.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: msi pm cap_list
configuration: latency=0
resources: iomemory:600-5ff memory:603d1b2000-603d1b2fff
*-usb:0
description: USB controller
product: Tiger Lake-LP Thunderbolt 4 USB Controller
vendor: Intel Corporation
physical id: d
bus info: pci@0000:00:0d.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: pm msi xhci bus_master cap_list
configuration: driver=xhci_hcd latency=0
resources: iomemory:600-5ff irq:130 memory:603d190000-603d19ffff
*-usbhost:0
product: xHCI Host Controller
vendor: Linux 5.15.0-88-generic xhci-hcd
physical id: 0
bus info: usb@1
logical name: usb1
version: 5.15
capabilities: usb-2.00
configuration: driver=hub slots=1 speed=480Mbit/s
*-usbhost:1
product: xHCI Host Controller
vendor: Linux 5.15.0-88-generic xhci-hcd
physical id: 1
bus info: usb@2
logical name: usb2
version: 5.15
capabilities: usb-3.10
configuration: driver=hub slots=4 speed=10000Mbit/s
*-usb:1
description: USB controller
product: Tiger Lake-LP Thunderbolt 4 NHI #0
vendor: Intel Corporation
physical id: d.2
bus info: pci@0000:00:0d.2
version: 01
width: 64 bits
clock: 33MHz
capabilities: pm msi msix usb4_host_interface bus_master cap_list
configuration: driver=thunderbolt latency=0
resources: iomemory:600-5ff iomemory:600-5ff irq:16 memory:603d140000-603d17ffff memory:603d1b1000-603d1b1fff
*-usb:2
description: USB controller
product: Tiger Lake-LP Thunderbolt 4 NHI #1
vendor: Intel Corporation
physical id: d.3
bus info: pci@0000:00:0d.3
version: 01
width: 64 bits
clock: 33MHz
capabilities: pm msi msix usb4_host_interface bus_master cap_list
configuration: driver=thunderbolt latency=0
resources: iomemory:600-5ff iomemory:600-5ff irq:16 memory:603d100000-603d13ffff memory:603d1b0000-603d1b0fff
*-usb:3
description: USB controller
product: Tiger Lake-LP USB 3.2 Gen 2x1 xHCI Host Controller
vendor: Intel Corporation
physical id: 14
bus info: pci@0000:00:14.0
version: 20
width: 64 bits
clock: 33MHz
capabilities: pm msi xhci bus_master cap_list
configuration: driver=xhci_hcd latency=0
resources: iomemory:600-5ff irq:148 memory:603d180000-603d18ffff
*-usbhost:0
product: xHCI Host Controller
vendor: Linux 5.15.0-88-generic xhci-hcd
physical id: 0
bus info: usb@3
logical name: usb3
version: 5.15
capabilities: usb-2.00
configuration: driver=hub slots=12 speed=480Mbit/s
*-usb
description: Bluetooth wireless interface
product: AX201 Bluetooth
vendor: Intel Corp.
physical id: a
bus info: usb@3:a
version: 0.02
capabilities: bluetooth usb-2.01
configuration: driver=btusb maxpower=100mA speed=12Mbit/s
*-usbhost:1
product: xHCI Host Controller
vendor: Linux 5.15.0-88-generic xhci-hcd
physical id: 1
bus info: usb@4
logical name: usb4
version: 5.15
capabilities: usb-3.10
configuration: driver=hub slots=4 speed=10000Mbit/s
*-memory UNCLAIMED
description: RAM memory
product: Tiger Lake-LP Shared SRAM
vendor: Intel Corporation
physical id: 14.2
bus info: pci@0000:00:14.2
version: 20
width: 64 bits
clock: 33MHz (30.3ns)
capabilities: pm cap_list
configuration: latency=0
resources: iomemory:600-5ff iomemory:600-5ff memory:603d1a8000-603d1abfff memory:603d1af000-603d1affff
*-network DISABLED
description: Wireless interface
product: Wi-Fi 6 AX201
vendor: Intel Corporation
physical id: 14.3
bus info: pci@0000:00:14.3
logical name: wlo1
version: 20
serial: c4:75:ab:93:65:84
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical wireless
configuration: broadcast=yes driver=iwlwifi driverversion=5.15.0-88-generic firmware=66.f1c864e0.0 QuZ-a0-hr-b0-66.u latency=0 link=no multicast=yes wireless=IEEE 802.11
resources: iomemory:600-5ff irq:19 memory:603d1a4000-603d1a7fff
*-serial:0
description: Serial bus controller
product: Tiger Lake-LP Serial IO I2C Controller #0
vendor: Intel Corporation
physical id: 15
bus info: pci@0000:00:15.0
version: 20
width: 64 bits
clock: 33MHz
capabilities: pm bus_master cap_list
configuration: driver=intel-lpss latency=0
resources: irq:27 memory:4017000000-4017000fff
*-communication
description: Communication controller
product: Tiger Lake-LP Management Engine Interface
vendor: Intel Corporation
physical id: 16
bus info: pci@0000:00:16.0
version: 20
width: 64 bits
clock: 33MHz
capabilities: pm msi bus_master cap_list
configuration: driver=mei_me latency=0
resources: iomemory:600-5ff irq:180 memory:603d1ad000-603d1adfff
*-sata
description: SATA controller
product: Intel Corporation
vendor: Intel Corporation
physical id: 17
bus info: pci@0000:00:17.0
logical name: scsi0
version: 20
width: 32 bits
clock: 66MHz
capabilities: sata msi pm ahci_1.0 bus_master cap_list emulated
configuration: driver=ahci latency=0
resources: irq:162 memory:6a700000-6a701fff memory:6a703000-6a7030ff ioport:3090(size=8) ioport:3080(size=4) ioport:3060(size=32) memory:6a702000-6a7027ff
*-disk
description: ATA Disk
product: Samsung SSD 860
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: 1B6Q
serial: 43dxr3-TC3T-eYel-ksUD-VZCs-qqMh-qtetIo
size: 1863GiB
capacity: 1863GiB
capabilities: lvm2
configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512
*-pci:3
description: PCI bridge
product: Intel Corporation
vendor: Intel Corporation
physical id: 1c
bus info: pci@0000:00:1c.0
version: 20
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:127 memory:6a500000-6a5fffff
*-generic
description: MMC Host
product: RTS525A PCI Express Card Reader
vendor: Realtek Semiconductor Co., Ltd.
physical id: 0
bus info: pci@0000:58:00.0
logical name: mmc0
version: 01
width: 32 bits
clock: 33MHz
capabilities: pm msi pciexpress bus_master cap_list
configuration: driver=rtsx_pci latency=0
resources: irq:147 memory:6a500000-6a500fff
*-pci:4
description: PCI bridge
product: Intel Corporation
vendor: Intel Corporation
physical id: 1c.6
bus info: pci@0000:00:1c.6
version: 20
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:128 memory:6a200000-6a4fffff
*-network
description: Ethernet interface
product: Ethernet Controller I225-V
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:59:00.0
logical name: enp89s0
version: 03
serial: 88:ae:dd:0d:c9:af
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=igc driverversion=5.15.0-88-generic duplex=full firmware=1085:8770 ip=10.0.0.110 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:18 memory:6a300000-6a3fffff memory:6a400000-6a403fff memory:6a200000-6a2fffff
*-isa
description: ISA bridge
product: Tiger Lake-LP LPC Controller
vendor: Intel Corporation
physical id: 1f
bus info: pci@0000:00:1f.0
version: 20
width: 32 bits
clock: 33MHz
capabilities: isa bus_master
configuration: latency=0
*-pnp00:00
product: PnP device PNP0c02
physical id: 0
capabilities: pnp
configuration: driver=system
*-pnp00:01
product: PnP device PNP0c02
physical id: 1
capabilities: pnp
configuration: driver=system
*-pnp00:02
product: PnP device PNP0c02
physical id: 2
capabilities: pnp
configuration: driver=system
*-pnp00:03
product: PnP device PNP0c02
physical id: 3
capabilities: pnp
configuration: driver=system
*-pnp00:04
product: PnP device PNP0c02
physical id: 4
capabilities: pnp
configuration: driver=system
*-pnp00:05
product: PnP device PNP0c02
physical id: 5
capabilities: pnp
configuration: driver=system
*-pnp00:06
product: PnP device PNP0c02
physical id: 6
capabilities: pnp
configuration: driver=system
*-pnp00:07
product: PnP device PNP0c02
physical id: 7
capabilities: pnp
configuration: driver=system
*-multimedia
description: Audio device
product: Tiger Lake-LP Smart Sound Technology Audio Controller
vendor: Intel Corporation
physical id: 1f.3
bus info: pci@0000:00:1f.3
logical name: card0
logical name: /dev/snd/controlC0
logical name: /dev/snd/hwC0D2
logical name: /dev/snd/pcmC0D10p
logical name: /dev/snd/pcmC0D11p
logical name: /dev/snd/pcmC0D12p
logical name: /dev/snd/pcmC0D13p
logical name: /dev/snd/pcmC0D14p
logical name: /dev/snd/pcmC0D15p
logical name: /dev/snd/pcmC0D16p
logical name: /dev/snd/pcmC0D17p
logical name: /dev/snd/pcmC0D3p
logical name: /dev/snd/pcmC0D7p
logical name: /dev/snd/pcmC0D8p
logical name: /dev/snd/pcmC0D9p
version: 20
width: 64 bits
clock: 33MHz
capabilities: pm msi bus_master cap_list
configuration: driver=snd_hda_intel latency=32
resources: iomemory:600-5ff iomemory:600-5ff irq:16 memory:603d1a0000-603d1a3fff memory:603d000000-603d0fffff
*-input:0
product: HDA Intel PCH HDMI/DP,pcm=12
physical id: 0
logical name: input10
logical name: /dev/input/event10
*-input:1
product: HDA Intel PCH HDMI/DP,pcm=13
physical id: 1
logical name: input11
logical name: /dev/input/event11
*-input:2
product: HDA Intel PCH HDMI/DP,pcm=14
physical id: 2
logical name: input12
logical name: /dev/input/event12
*-input:3
product: HDA Intel PCH HDMI/DP,pcm=15
physical id: 3
logical name: input13
logical name: /dev/input/event13
*-input:4
product: HDA Intel PCH HDMI/DP,pcm=16
physical id: 4
logical name: input14
logical name: /dev/input/event14
*-input:5
product: HDA Intel PCH HDMI/DP,pcm=17
physical id: 5
logical name: input15
logical name: /dev/input/event15
*-input:6
product: HDA Intel PCH HDMI/DP,pcm=3
physical id: 6
logical name: input4
logical name: /dev/input/event4
*-input:7
product: HDA Intel PCH HDMI/DP,pcm=7
physical id: 7
logical name: input5
logical name: /dev/input/event5
*-input:8
product: HDA Intel PCH HDMI/DP,pcm=8
physical id: 8
logical name: input6
logical name: /dev/input/event6
*-input:9
product: HDA Intel PCH HDMI/DP,pcm=9
physical id: 9
logical name: input7
logical name: /dev/input/event7
*-input:10
product: HDA Intel PCH HDMI/DP,pcm=10
physical id: a
logical name: input8
logical name: /dev/input/event8
*-input:11
product: HDA Intel PCH HDMI/DP,pcm=11
physical id: b
logical name: input9
logical name: /dev/input/event9
*-serial:1
description: SMBus
product: Tiger Lake-LP SMBus Controller
vendor: Intel Corporation
physical id: 1f.4
bus info: pci@0000:00:1f.4
version: 20
width: 64 bits
clock: 33MHz
configuration: driver=i801_smbus latency=0
resources: iomemory:600-5ff irq:16 memory:603d1ac000-603d1ac0ff ioport:efa0(size=32)
*-serial:2 UNCLAIMED
description: Serial bus controller
product: Tiger Lake-LP SPI Controller
vendor: Intel Corporation
physical id: 1f.5
bus info: pci@0000:00:1f.5
version: 20
width: 32 bits
clock: 33MHz
configuration: latency=0
resources: memory:4f800000-4f800fff
*-power:0 UNCLAIMED
physical id: 1
capacity: 32768mWh
*-power:1 UNCLAIMED
physical id: 2
capacity: 32768mWh
*-power:2 UNCLAIMED
physical id: 3
capacity: 32768mWh
*-input:0
product: Sleep Button
physical id: 4
logical name: input0
logical name: /dev/input/event0
capabilities: platform
*-input:1
product: Power Button
physical id: 5
logical name: input1
logical name: /dev/input/event1
capabilities: platform
*-input:2
product: Power Button
physical id: 6
logical name: input2
logical name: /dev/input/event2
capabilities: platform
*-input:3
product: Video Bus
physical id: 7
logical name: input3
logical name: /dev/input/event3
capabilities: platform
Just the standard ethereum staking stack:
- execution (geth)
- consensus (lighthouse)
- validators (lighthouse)
- block building (mev-boost)
Could you try to turn your smart features off?
sudo smartctl --smart=off /dev/sda
https://serverfault.com/questions/641505/calling-sync-fsync-slows-io-after-30-minutes-uptime
It could be that your disk is trying to run some checks in the background which makes all fsync's slow
I'll give that a shot, sure. However, I'm not confident that is the issue here.
For example, during the course of this conversation (after I posted the debug logs), I rolled back to v1.13.4, and all of the elapsed times for the Imported new potential chain segment logs are around 50-100ms, as is typical on my machine.
It's only when running v1.13.5 that I'm seeing these slow elapsed times.
@adamgall In this v1.13.5, we have a fix to prevent some corruptions after power failure, which basically has additional two file.sync operation for each block.
We internally test it a bit and the overhead for these sync operations is pretty trivial, and apparently it's very slow for you which also surprises us a bit.
I've upgraded to geth v1.13.5 again, disabled the smart features on my Samsung SSD (/dev/sda, which is mounted at geth's ancient directory), and am now monitoring with debug.vmodule("trie/*=5"). I'll be back in ~10 mins with those logs.
Seeing similar behavior: periods of normal elapsed times, followed by periods of long elapsed times, and repeat.
INFO [11-16|14:31:11.631] Starting Geth on Ethereum mainnet...
INFO [11-16|14:31:11.632] Bumping default cache on mainnet provided=1024 updated=4096
INFO [11-16|14:31:11.633] Maximum peer count ETH=50 LES=0 total=50
INFO [11-16|14:31:11.634] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [11-16|14:31:11.634] Using pebble as db engine
INFO [11-16|14:31:11.636] Set global gas cap cap=50,000,000
INFO [11-16|14:31:11.636] Initializing the KZG library backend=gokzg
INFO [11-16|14:31:11.660] Allocated trie memory caches clean=614.00MiB dirty=1024.00MiB
INFO [11-16|14:31:11.830] Using pebble as the backing database
INFO [11-16|14:31:11.830] Allocated cache and file handles database=/var/lib/geth/geth/chaindata cache=2.00GiB handles=262,144
INFO [11-16|14:31:15.633] Found legacy ancient chain path location=/var/lib/geth/geth/chaindata/ancient
INFO [11-16|14:31:15.686] Opened ancient database database=/var/lib/geth/geth/chaindata/ancient readonly=false
INFO [11-16|14:31:15.710] State scheme set by user scheme=path
INFO [11-16|14:31:15.832] Initialising Ethereum protocol network=1 dbversion=8
WARN [11-16|14:31:15.832] Sanitizing invalid node buffer size provided=1024.00MiB updated=256.00MiB
INFO [11-16|14:31:17.194] Opened ancient database database=/var/lib/geth/geth/chaindata/ancient/state readonly=false
WARN [11-16|14:31:17.203] Path-based state scheme is an experimental feature
INFO [11-16|14:31:17.204]
INFO [11-16|14:31:17.204] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [11-16|14:31:17.204] Chain ID: 1 (mainnet)
INFO [11-16|14:31:17.204] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [11-16|14:31:17.204]
INFO [11-16|14:31:17.204] Pre-Merge hard forks (block based):
INFO [11-16|14:31:17.204] - Homestead: #1150000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [11-16|14:31:17.204] - DAO Fork: #1920000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [11-16|14:31:17.204] - Tangerine Whistle (EIP 150): #2463000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [11-16|14:31:17.204] - Spurious Dragon/1 (EIP 155): #2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [11-16|14:31:17.204] - Spurious Dragon/2 (EIP 158): #2675000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [11-16|14:31:17.204] - Byzantium: #4370000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [11-16|14:31:17.204] - Constantinople: #7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [11-16|14:31:17.204] - Petersburg: #7280000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [11-16|14:31:17.204] - Istanbul: #9069000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [11-16|14:31:17.204] - Muir Glacier: #9200000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [11-16|14:31:17.204] - Berlin: #12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [11-16|14:31:17.204] - London: #12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [11-16|14:31:17.204] - Arrow Glacier: #13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [11-16|14:31:17.204] - Gray Glacier: #15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [11-16|14:31:17.204]
INFO [11-16|14:31:17.204] Merge configured:
INFO [11-16|14:31:17.204] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [11-16|14:31:17.204] - Network known to be merged: true
INFO [11-16|14:31:17.204] - Total terminal difficulty: 58750000000000000000000
INFO [11-16|14:31:17.204]
INFO [11-16|14:31:17.204] Post-Merge hard forks (timestamp based):
INFO [11-16|14:31:17.204] - Shanghai: @1681338455 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [11-16|14:31:17.204]
INFO [11-16|14:31:17.204] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [11-16|14:31:17.204]
INFO [11-16|14:31:17.206] Loaded most recent local block number=18,585,057 hash=06b145..0fa63a td=58,750,003,716,598,352,816,469 age=18s
INFO [11-16|14:31:17.206] Loaded most recent local finalized block number=18,584,977 hash=e06b23..605add td=58,750,003,716,598,352,816,469 age=16m30s
INFO [11-16|14:31:17.207] Loaded last snap-sync pivot marker number=18,180,334
INFO [11-16|14:31:17.563] Initialized transaction indexer limit=2,350,000
INFO [11-16|14:31:17.563] Loaded local transaction journal transactions=0 dropped=0
INFO [11-16|14:31:17.563] Regenerated local transaction journal transactions=0 accounts=0
WARN [11-16|14:31:17.569] Switch sync mode from snap sync to full sync reason="snap sync complete"
INFO [11-16|14:31:17.569] Chain post-merge, sync via beacon client
INFO [11-16|14:31:17.569] Gasprice oracle is ignoring threshold set threshold=2
WARN [11-16|14:31:17.598] Engine API enabled protocol=eth
INFO [11-16|14:31:17.598] Starting peer-to-peer node instance=Geth/v1.13.5-stable-916d6a44/linux-amd64/go1.20.2
INFO [11-16|14:31:17.637] New local node record seq=1,681,344,381,530 id=a0146bc537df9cf1 ip=127.0.0.1 udp=30303 tcp=30303
INFO [11-16|14:31:17.638] Started P2P networking self=enode://2acdf0bc4b77479c8f0b3dcfd22e754e977a969317d4dc676b25efdd0ccc217a671deb2eb9dc491cc7ec7ec31857e2745e6a772c80ffc31f0b4ad4db7a111547@127.0.0.1:30303
INFO [11-16|14:31:17.638] IPC endpoint opened url=/var/lib/geth/geth.ipc
INFO [11-16|14:31:17.638] Loaded JWT secret file path=/var/lib/ethereum-shared/jwt-primary.hex crc32=0x1b524a01
INFO [11-16|14:31:17.639] HTTP server started endpoint=127.0.0.1:8545 auth=false prefix= cors= vhosts=localhost
INFO [11-16|14:31:17.639] WebSocket enabled url=ws://127.0.0.1:8551
INFO [11-16|14:31:17.639] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost
INFO [11-16|14:31:18.686] New local node record seq=1,681,344,381,531 id=a0146bc537df9cf1 ip=174.61.110.113 udp=30303 tcp=30303
INFO [11-16|14:31:20.293] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [11-16|14:31:20.739] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-IP1"
INFO [11-16|14:31:23.262] Imported new potential chain segment number=18,585,058 hash=d8ef6e..ac06e9 blocks=1 txs=157 mgas=12.379 elapsed=333.099ms mgasps=37.164 snapdiffs=6.91MiB triediffs=178.82MiB triedirty=37.07MiB
INFO [11-16|14:31:23.407] Chain head was updated number=18,585,058 hash=d8ef6e..ac06e9 root=bdcde9..93ffa2 elapsed=2.528052ms
INFO [11-16|14:31:27.390] Imported new potential chain segment number=18,585,059 hash=911da7..4cdaa1 blocks=1 txs=150 mgas=17.663 elapsed=2.328s mgasps=7.584 snapdiffs=6.96MiB triediffs=178.93MiB triedirty=37.50MiB
INFO [11-16|14:31:27.546] Chain head was updated number=18,585,059 hash=911da7..4cdaa1 root=5be5d2..328d32 elapsed=3.75658ms
INFO [11-16|14:31:27.642] Looking for peers peercount=3 tried=169 static=0
INFO [11-16|14:31:38.777] Imported new potential chain segment number=18,585,060 hash=764372..54a13b blocks=1 txs=144 mgas=11.830 elapsed=2.019s mgasps=5.857 snapdiffs=7.00MiB triediffs=178.47MiB triedirty=38.05MiB
INFO [11-16|14:31:38.924] Chain head was updated number=18,585,060 hash=764372..54a13b root=b2d889..fa3303 elapsed=2.587397ms
INFO [11-16|14:31:51.139] Imported new potential chain segment number=18,585,061 hash=6d7d90..c6a4dc blocks=1 txs=153 mgas=12.974 elapsed=2.066s mgasps=6.278 snapdiffs=7.03MiB triediffs=178.88MiB triedirty=38.35MiB
INFO [11-16|14:31:51.329] Chain head was updated number=18,585,061 hash=6d7d90..c6a4dc root=dd3e47..d41882 elapsed=2.310415ms
INFO [11-16|14:32:03.247] Imported new potential chain segment number=18,585,062 hash=c32c11..07bd20 blocks=1 txs=151 mgas=11.090 elapsed=2.022s mgasps=5.483 snapdiffs=7.07MiB triediffs=178.15MiB triedirty=38.93MiB
INFO [11-16|14:32:03.389] Chain head was updated number=18,585,062 hash=c32c11..07bd20 root=ec421c..7909c0 elapsed=3.342737ms
INFO [11-16|14:32:14.582] Imported new potential chain segment number=18,585,063 hash=a993d4..ade5ae blocks=1 txs=207 mgas=15.554 elapsed=2.000s mgasps=7.776 snapdiffs=7.12MiB triediffs=178.47MiB triedirty=39.42MiB
INFO [11-16|14:32:14.692] Chain head was updated number=18,585,063 hash=a993d4..ade5ae root=2bec9d..107676 elapsed=3.114937ms
INFO [11-16|14:32:26.756] Imported new potential chain segment number=18,585,064 hash=0e8249..ea41be blocks=1 txs=114 mgas=16.937 elapsed=1.720s mgasps=9.844 snapdiffs=7.14MiB triediffs=178.02MiB triedirty=39.79MiB
INFO [11-16|14:32:26.941] Chain head was updated number=18,585,064 hash=0e8249..ea41be root=47923d..b10300 elapsed=5.034499ms
INFO [11-16|14:32:38.761] Imported new potential chain segment number=18,585,065 hash=07f60d..10605f blocks=1 txs=143 mgas=22.901 elapsed=168.842ms mgasps=135.634 snapdiffs=7.18MiB triediffs=177.87MiB triedirty=40.05MiB
INFO [11-16|14:32:38.908] Chain head was updated number=18,585,065 hash=07f60d..10605f root=a60d11..86edbf elapsed=9.018851ms
INFO [11-16|14:32:48.880] Imported new potential chain segment number=18,585,066 hash=9e1440..4b5220 blocks=1 txs=178 mgas=21.794 elapsed=317.586ms mgasps=68.622 snapdiffs=7.22MiB triediffs=178.36MiB triedirty=40.39MiB
INFO [11-16|14:32:48.978] Chain head was updated number=18,585,066 hash=9e1440..4b5220 root=67a2ef..e3975b elapsed=2.688092ms
INFO [11-16|14:33:00.163] Imported new potential chain segment number=18,585,067 hash=a31a64..a8f364 blocks=1 txs=150 mgas=13.833 elapsed=112.239ms mgasps=123.249 snapdiffs=7.25MiB triediffs=178.04MiB triedirty=40.72MiB
INFO [11-16|14:33:00.250] Chain head was updated number=18,585,067 hash=a31a64..a8f364 root=ce5c7b..be9626 elapsed=1.855188ms
INFO [11-16|14:33:12.828] Imported new potential chain segment number=18,585,068 hash=b2db19..0efe44 blocks=1 txs=284 mgas=15.644 elapsed=96.251ms mgasps=162.531 snapdiffs=7.29MiB triediffs=178.68MiB triedirty=41.05MiB
INFO [11-16|14:33:12.922] Chain head was updated number=18,585,068 hash=b2db19..0efe44 root=03ab62..d9bbb9 elapsed=2.374607ms
INFO [11-16|14:33:26.124] Imported new potential chain segment number=18,585,069 hash=5c7ccf..f4a5f8 blocks=1 txs=117 mgas=9.956 elapsed=1.732s mgasps=5.746 snapdiffs=7.32MiB triediffs=178.40MiB triedirty=41.34MiB
INFO [11-16|14:33:26.242] Chain head was updated number=18,585,069 hash=5c7ccf..f4a5f8 root=3b90ef..039c8c elapsed=2.515672ms
INFO [11-16|14:33:39.289] Imported new potential chain segment number=18,585,070 hash=8bc5b0..301c1e blocks=1 txs=187 mgas=20.334 elapsed=2.068s mgasps=9.833 snapdiffs=7.37MiB triediffs=179.43MiB triedirty=41.56MiB
INFO [11-16|14:33:39.414] Chain head was updated number=18,585,070 hash=8bc5b0..301c1e root=d23646..8361fd elapsed=4.244974ms
INFO [11-16|14:33:51.114] Imported new potential chain segment number=18,585,071 hash=427b9c..3f7c68 blocks=1 txs=128 mgas=13.177 elapsed=1.989s mgasps=6.622 snapdiffs=7.40MiB triediffs=178.51MiB triedirty=42.14MiB
INFO [11-16|14:33:51.252] Chain head was updated number=18,585,071 hash=427b9c..3f7c68 root=153c3a..b2e0d8 elapsed=3.199256ms
WARN [11-16|14:33:57.472] Served eth_coinbase reqid=3 duration="16.264µs" err="etherbase must be explicitly specified"
INFO [11-16|14:34:02.694] Imported new potential chain segment number=18,585,072 hash=abfe47..564242 blocks=1 txs=131 mgas=10.713 elapsed=1.913s mgasps=5.599 snapdiffs=7.44MiB triediffs=178.21MiB triedirty=42.49MiB
INFO [11-16|14:34:04.361] Chain head was updated number=18,585,072 hash=abfe47..564242 root=f00f83..8e185b elapsed=3.386374ms
[turned on trie debug logging here via debug.vmodule("trie/*=5")]
DEBUG[11-16|14:34:12.157] Created new diff layer id=404,739 block=18,585,073 nodes=4448 size=1.68MiB
DEBUG[11-16|14:34:14.068] Stored state history id=404,611 block=18,584,945 data=15.60KiB index=24.91KiB elapsed=1.911s
DEBUG[11-16|14:34:14.071] Pruned state history items=1 tailid=314,612
INFO [11-16|14:34:14.071] Imported new potential chain segment number=18,585,073 hash=5aa799..424a00 blocks=1 txs=192 mgas=14.869 elapsed=1.985s mgasps=7.489 snapdiffs=7.48MiB triediffs=178.52MiB triedirty=42.85MiB
INFO [11-16|14:34:14.226] Chain head was updated number=18,585,073 hash=5aa799..424a00 root=931659..d9921e elapsed=3.166036ms
DEBUG[11-16|14:34:24.867] Created new diff layer id=404,740 block=18,585,074 nodes=2704 size=1.04MiB
DEBUG[11-16|14:34:25.841] Stored state history id=404,612 block=18,584,946 data=15.61KiB index=28.02KiB elapsed=974.569ms
DEBUG[11-16|14:34:25.844] Pruned state history items=1 tailid=314,613
INFO [11-16|14:34:25.844] Imported new potential chain segment number=18,585,074 hash=e9641e..e880f4 blocks=1 txs=125 mgas=10.632 elapsed=1.025s mgasps=10.368 snapdiffs=7.51MiB triediffs=178.10MiB triedirty=43.22MiB
INFO [11-16|14:34:25.998] Chain head was updated number=18,585,074 hash=e9641e..e880f4 root=2da6ce..88db0d elapsed=4.84476ms
DEBUG[11-16|14:34:37.474] Created new diff layer id=404,741 block=18,585,075 nodes=5067 size=1.86MiB
DEBUG[11-16|14:34:37.590] Stored state history id=404,613 block=18,584,947 data=11.15KiB index=16.78KiB elapsed=116.347ms
DEBUG[11-16|14:34:37.593] Pruned state history items=1 tailid=314,614
INFO [11-16|14:34:37.593] Imported new potential chain segment number=18,585,075 hash=a09416..2325ae blocks=1 txs=194 mgas=20.380 elapsed=204.642ms mgasps=99.589 snapdiffs=7.56MiB triediffs=179.02MiB triedirty=43.45MiB
INFO [11-16|14:34:37.715] Chain head was updated number=18,585,075 hash=a09416..2325ae root=8e7144..55464a elapsed=2.555478ms
DEBUG[11-16|14:34:48.669] Created new diff layer id=404,742 block=18,585,076 nodes=2749 size=1.04MiB
DEBUG[11-16|14:34:48.767] Stored state history id=404,614 block=18,584,948 data=14.14KiB index=22.28KiB elapsed=98.383ms
DEBUG[11-16|14:34:48.769] Pruned state history items=1 tailid=314,615
INFO [11-16|14:34:48.769] Imported new potential chain segment number=18,585,076 hash=387149..4f3894 blocks=1 txs=136 mgas=13.120 elapsed=164.702ms mgasps=79.658 snapdiffs=7.59MiB triediffs=178.82MiB triedirty=43.74MiB
INFO [11-16|14:34:48.887] Chain head was updated number=18,585,076 hash=387149..4f3894 root=563761..ec607e elapsed=2.754364ms
DEBUG[11-16|14:35:00.651] Created new diff layer id=404,743 block=18,585,077 nodes=4157 size=1.53MiB
DEBUG[11-16|14:35:00.717] Stored state history id=404,615 block=18,584,949 data=18.00KiB index=31.55KiB elapsed=65.596ms
DEBUG[11-16|14:35:00.720] Pruned state history items=1 tailid=314,616
INFO [11-16|14:35:00.720] Imported new potential chain segment number=18,585,077 hash=fb7dba..c89ad0 blocks=1 txs=157 mgas=14.988 elapsed=176.867ms mgasps=84.742 snapdiffs=7.63MiB triediffs=178.58MiB triedirty=44.21MiB
INFO [11-16|14:35:00.850] Chain head was updated number=18,585,077 hash=fb7dba..c89ad0 root=b54900..7f5f27 elapsed=2.486076ms
DEBUG[11-16|14:35:13.919] Created new diff layer id=404,744 block=18,585,078 nodes=4046 size=1.52MiB
DEBUG[11-16|14:35:14.018] Stored state history id=404,616 block=18,584,950 data=14.67KiB index=23.83KiB elapsed=99.391ms
DEBUG[11-16|14:35:14.021] Pruned state history items=1 tailid=314,617
INFO [11-16|14:35:14.021] Imported new potential chain segment number=18,585,078 hash=cde76f..dd5c5a blocks=1 txs=190 mgas=14.898 elapsed=175.663ms mgasps=84.810 snapdiffs=7.67MiB triediffs=178.79MiB triedirty=44.53MiB
INFO [11-16|14:35:14.217] Chain head was updated number=18,585,078 hash=cde76f..dd5c5a root=3102db..d5ff44 elapsed=2.975144ms
DEBUG[11-16|14:35:25.290] Created new diff layer id=404,745 block=18,585,079 nodes=3810 size=1.40MiB
DEBUG[11-16|14:35:27.203] Stored state history id=404,617 block=18,584,951 data=14.16KiB index=22.95KiB elapsed=1.912s
DEBUG[11-16|14:35:27.205] Pruned state history items=1 tailid=314,618
INFO [11-16|14:35:27.205] Imported new potential chain segment number=18,585,079 hash=a6868a..434e5f blocks=1 txs=162 mgas=18.618 elapsed=1.976s mgasps=9.419 snapdiffs=7.71MiB triediffs=178.93MiB triedirty=44.86MiB
INFO [11-16|14:35:27.385] Chain head was updated number=18,585,079 hash=a6868a..434e5f root=3bc26a..540c38 elapsed=7.808752ms
DEBUG[11-16|14:35:36.912] Created new diff layer id=404,746 block=18,585,080 nodes=3159 size=1.20MiB
DEBUG[11-16|14:35:38.898] Stored state history id=404,618 block=18,584,952 data=18.22KiB index=28.99KiB elapsed=1.986s
DEBUG[11-16|14:35:38.900] Pruned state history items=1 tailid=314,619
INFO [11-16|14:35:38.900] Imported new potential chain segment number=18,585,080 hash=d2a9af..ade734 blocks=1 txs=180 mgas=11.092 elapsed=2.023s mgasps=5.482 snapdiffs=7.74MiB triediffs=178.55MiB triedirty=45.27MiB
INFO [11-16|14:35:39.044] Chain head was updated number=18,585,080 hash=d2a9af..ade734 root=da8167..a5bed3 elapsed=2.311374ms
DEBUG[11-16|14:35:48.307] Created new diff layer id=404,747 block=18,585,081 nodes=3727 size=1.39MiB
DEBUG[11-16|14:35:50.250] Stored state history id=404,619 block=18,584,953 data=11.46KiB index=17.83KiB elapsed=1.942s
DEBUG[11-16|14:35:50.252] Pruned state history items=1 tailid=314,620
INFO [11-16|14:35:50.252] Imported new potential chain segment number=18,585,081 hash=284ed4..b58e00 blocks=1 txs=192 mgas=14.885 elapsed=2.011s mgasps=7.401 snapdiffs=7.78MiB triediffs=178.95MiB triedirty=45.50MiB
INFO [11-16|14:35:50.408] Chain head was updated number=18,585,081 hash=284ed4..b58e00 root=02528c..7793d8 elapsed=3.6645ms
DEBUG[11-16|14:36:00.433] Created new diff layer id=404,748 block=18,585,082 nodes=4434 size=1.63MiB
DEBUG[11-16|14:36:02.296] Stored state history id=404,620 block=18,584,954 data=20.81KiB index=33.54KiB elapsed=1.863s
DEBUG[11-16|14:36:02.298] Pruned state history items=1 tailid=314,621
INFO [11-16|14:36:02.298] Imported new potential chain segment number=18,585,082 hash=139623..d435aa blocks=1 txs=185 mgas=15.703 elapsed=2.013s mgasps=7.801 snapdiffs=7.83MiB triediffs=178.80MiB triedirty=45.94MiB
INFO [11-16|14:36:02.445] Chain head was updated number=18,585,082 hash=139623..d435aa root=97fd23..ce6686 elapsed=3.221268ms
DEBUG[11-16|14:36:12.586] Created new diff layer id=404,749 block=18,585,083 nodes=4298 size=1.62MiB
DEBUG[11-16|14:36:14.464] Stored state history id=404,621 block=18,584,955 data=18.00KiB index=31.20KiB elapsed=1.878s
DEBUG[11-16|14:36:14.467] Pruned state history items=1 tailid=314,622
INFO [11-16|14:36:14.467] Imported new potential chain segment number=18,585,083 hash=3fdfdf..0a65cb blocks=1 txs=211 mgas=15.127 elapsed=1.937s mgasps=7.808 snapdiffs=7.87MiB triediffs=178.76MiB triedirty=46.40MiB
INFO [11-16|14:36:14.691] Chain head was updated number=18,585,083 hash=3fdfdf..0a65cb root=d7d733..a46c40 elapsed=3.021016ms
DEBUG[11-16|14:36:25.239] Created new diff layer id=404,750 block=18,585,084 nodes=3667 size=1.37MiB
DEBUG[11-16|14:36:25.741] Stored state history id=404,622 block=18,584,956 data=15.52KiB index=25.37KiB elapsed=501.656ms
DEBUG[11-16|14:36:25.743] Pruned state history items=1 tailid=314,623
INFO [11-16|14:36:25.743] Imported new potential chain segment number=18,585,084 hash=cda319..b502f0 blocks=1 txs=220 mgas=13.418 elapsed=567.760ms mgasps=23.633 snapdiffs=7.91MiB triediffs=178.79MiB triedirty=46.70MiB
INFO [11-16|14:36:25.945] Chain head was updated number=18,585,084 hash=cda319..b502f0 root=fe6dd1..a1246b elapsed=3.958675ms
DEBUG[11-16|14:36:37.202] Created new diff layer id=404,751 block=18,585,085 nodes=4125 size=1.55MiB
DEBUG[11-16|14:36:37.319] Stored state history id=404,623 block=18,584,957 data=12.91KiB index=23.34KiB elapsed=117.143ms
DEBUG[11-16|14:36:37.321] Pruned state history items=1 tailid=314,624
INFO [11-16|14:36:37.321] Imported new potential chain segment number=18,585,085 hash=8f4410..55fff4 blocks=1 txs=231 mgas=13.962 elapsed=184.088ms mgasps=75.842 snapdiffs=7.95MiB triediffs=179.07MiB triedirty=47.03MiB
INFO [11-16|14:36:37.439] Chain head was updated number=18,585,085 hash=8f4410..55fff4 root=1ad375..2b034d elapsed=5.738906ms
DEBUG[11-16|14:36:49.031] Created new diff layer id=404,752 block=18,585,086 nodes=4155 size=1.53MiB
DEBUG[11-16|14:36:49.142] Stored state history id=404,624 block=18,584,958 data=15.51KiB index=27.56KiB elapsed=110.556ms
DEBUG[11-16|14:36:49.144] Pruned state history items=1 tailid=314,625
INFO [11-16|14:36:49.144] Imported new potential chain segment number=18,585,086 hash=42dcc0..2cc877 blocks=1 txs=168 mgas=15.423 elapsed=190.000ms mgasps=81.174 snapdiffs=7.99MiB triediffs=179.26MiB triedirty=47.36MiB
INFO [11-16|14:36:49.259] Chain head was updated number=18,585,086 hash=42dcc0..2cc877 root=416384..4e3303 elapsed=2.356974ms
DEBUG[11-16|14:37:00.819] Created new diff layer id=404,753 block=18,585,087 nodes=3175 size=1.21MiB
DEBUG[11-16|14:37:00.848] Stored state history id=404,625 block=18,584,959 data=19.69KiB index=34.24KiB elapsed=28.901ms
DEBUG[11-16|14:37:00.850] Pruned state history items=1 tailid=314,626
INFO [11-16|14:37:00.850] Imported new potential chain segment number=18,585,087 hash=8a306b..ea4be2 blocks=1 txs=160 mgas=11.012 elapsed=89.042ms mgasps=123.671 snapdiffs=8.02MiB triediffs=178.63MiB triedirty=47.85MiB
INFO [11-16|14:37:00.966] Chain head was updated number=18,585,087 hash=8a306b..ea4be2 root=4c36f7..0b52db elapsed=1.902946ms
DEBUG[11-16|14:37:13.417] Created new diff layer id=404,754 block=18,585,088 nodes=4666 size=1.73MiB
DEBUG[11-16|14:37:13.506] Stored state history id=404,626 block=18,584,960 data=14.15KiB index=23.85KiB elapsed=88.832ms
DEBUG[11-16|14:37:13.508] Pruned state history items=1 tailid=314,627
INFO [11-16|14:37:13.508] Imported new potential chain segment number=18,585,088 hash=36625b..9b35f9 blocks=1 txs=198 mgas=18.902 elapsed=193.213ms mgasps=97.831 snapdiffs=8.07MiB triediffs=179.12MiB triedirty=48.14MiB
INFO [11-16|14:37:13.632] Chain head was updated number=18,585,088 hash=36625b..9b35f9 root=424711..fe3bc8 elapsed=2.716596ms
DEBUG[11-16|14:37:24.046] Created new diff layer id=404,755 block=18,585,089 nodes=3354 size=1.25MiB
DEBUG[11-16|14:37:25.953] Stored state history id=404,627 block=18,584,961 data=8.74KiB index=18.33KiB elapsed=1.907s
DEBUG[11-16|14:37:25.955] Pruned state history items=1 tailid=314,628
INFO [11-16|14:37:25.955] Imported new potential chain segment number=18,585,089 hash=a3ac5f..aae032 blocks=1 txs=144 mgas=15.464 elapsed=1.964s mgasps=7.872 snapdiffs=8.10MiB triediffs=179.45MiB triedirty=48.41MiB
INFO [11-16|14:37:26.124] Chain head was updated number=18,585,089 hash=a3ac5f..aae032 root=8ce270..9432d3 elapsed=2.996707ms
DEBUG[11-16|14:37:37.477] Created new diff layer id=404,756 block=18,585,090 nodes=3440 size=1.29MiB
DEBUG[11-16|14:37:39.471] Stored state history id=404,628 block=18,584,962 data=19.20KiB index=32.74KiB elapsed=1.993s
DEBUG[11-16|14:37:39.474] Pruned state history items=1 tailid=314,629
INFO [11-16|14:37:39.474] Imported new potential chain segment number=18,585,090 hash=0122d8..079f66 blocks=1 txs=149 mgas=12.094 elapsed=2.078s mgasps=5.818 snapdiffs=8.14MiB triediffs=179.08MiB triedirty=48.83MiB
INFO [11-16|14:37:39.618] Chain head was updated number=18,585,090 hash=0122d8..079f66 root=f59991..1dfe5c elapsed=2.748894ms
DEBUG[11-16|14:37:48.501] Created new diff layer id=404,757 block=18,585,091 nodes=2804 size=1.06MiB
DEBUG[11-16|14:37:50.446] Stored state history id=404,629 block=18,584,963 data=15.23KiB index=24.45KiB elapsed=1.944s
DEBUG[11-16|14:37:50.448] Pruned state history items=1 tailid=314,630
INFO [11-16|14:37:50.448] Imported new potential chain segment number=18,585,091 hash=d5b7d7..e1ac06 blocks=1 txs=120 mgas=10.645 elapsed=1.994s mgasps=5.337 snapdiffs=8.17MiB triediffs=178.81MiB triedirty=49.20MiB
INFO [11-16|14:37:50.639] Chain head was updated number=18,585,091 hash=d5b7d7..e1ac06 root=487efb..ca7a58 elapsed=2.337898ms
DEBUG[11-16|14:38:01.110] Created new diff layer id=404,758 block=18,585,092 nodes=4287 size=1.53MiB
DEBUG[11-16|14:38:02.975] Stored state history id=404,630 block=18,584,964 data=14.08KiB index=23.53KiB elapsed=1.865s
DEBUG[11-16|14:38:02.977] Pruned state history items=1 tailid=314,631
INFO [11-16|14:38:02.978] Imported new potential chain segment number=18,585,092 hash=1cb114..714fe1 blocks=1 txs=178 mgas=19.153 elapsed=1.939s mgasps=9.874 snapdiffs=8.22MiB triediffs=178.98MiB triedirty=49.51MiB
INFO [11-16|14:38:03.196] Chain head was updated number=18,585,092 hash=1cb114..714fe1 root=3b2ea7..258fc6 elapsed=2.733695ms
DEBUG[11-16|14:38:13.083] Created new diff layer id=404,759 block=18,585,093 nodes=5170 size=1.90MiB
DEBUG[11-16|14:38:15.009] Stored state history id=404,631 block=18,584,965 data=12.32KiB index=19.72KiB elapsed=1.925s
DEBUG[11-16|14:38:15.011] Pruned state history items=1 tailid=314,632
INFO [11-16|14:38:15.011] Imported new potential chain segment number=18,585,093 hash=7867e3..31c05a blocks=1 txs=333 mgas=20.359 elapsed=2.000s mgasps=10.177 snapdiffs=8.26MiB triediffs=179.78MiB triedirty=49.78MiB
INFO [11-16|14:38:15.155] Chain head was updated number=18,585,093 hash=7867e3..31c05a root=cb2dcb..fcdaa0 elapsed=5.201042ms
DEBUG[11-16|14:38:24.470] Created new diff layer id=404,760 block=18,585,094 nodes=2881 size=1.08MiB
DEBUG[11-16|14:38:24.586] Stored state history id=404,632 block=18,584,966 data=15.83KiB index=29.07KiB elapsed=116.076ms
DEBUG[11-16|14:38:24.589] Pruned state history items=1 tailid=314,633
INFO [11-16|14:38:24.589] Imported new potential chain segment number=18,585,094 hash=dce65b..a7273f blocks=1 txs=128 mgas=11.100 elapsed=177.371ms mgasps=62.581 snapdiffs=8.29MiB triediffs=179.39MiB triedirty=50.11MiB
INFO [11-16|14:38:24.707] Chain head was updated number=18,585,094 hash=dce65b..a7273f root=4a1ada..a0b499 elapsed=1.734002ms
DEBUG[11-16|14:38:36.835] Created new diff layer id=404,761 block=18,585,095 nodes=3947 size=1.46MiB
DEBUG[11-16|14:38:36.957] Stored state history id=404,633 block=18,584,967 data=15.21KiB index=24.86KiB elapsed=122.540ms
DEBUG[11-16|14:38:36.960] Pruned state history items=1 tailid=314,634
INFO [11-16|14:38:36.960] Imported new potential chain segment number=18,585,095 hash=0d6177..affde0 blocks=1 txs=166 mgas=14.126 elapsed=198.848ms mgasps=71.037 snapdiffs=8.34MiB triediffs=179.52MiB triedirty=50.43MiB
INFO [11-16|14:38:37.064] Chain head was updated number=18,585,095 hash=0d6177..affde0 root=875190..aee13b elapsed=2.997557ms
DEBUG[11-16|14:38:50.306] Created new diff layer id=404,762 block=18,585,096 nodes=2695 size=1.02MiB
DEBUG[11-16|14:38:50.375] Stored state history id=404,634 block=18,584,968 data=10.09KiB index=15.73KiB elapsed=68.397ms
DEBUG[11-16|14:38:50.383] Pruned state history items=1 tailid=314,635
INFO [11-16|14:38:50.383] Imported new potential chain segment number=18,585,096 hash=b1d277..b19074 blocks=1 txs=110 mgas=18.434 elapsed=148.694ms mgasps=123.973 snapdiffs=8.36MiB triediffs=179.59MiB triedirty=50.65MiB
INFO [11-16|14:38:50.508] Chain head was updated number=18,585,096 hash=b1d277..b19074 root=2e5f24..b39d5f elapsed=25.335527ms
DEBUG[11-16|14:39:02.062] Created new diff layer id=404,763 block=18,585,097 nodes=3720 size=1.39MiB
DEBUG[11-16|14:39:02.086] Stored state history id=404,635 block=18,584,969 data=28.17KiB index=46.38KiB elapsed=23.765ms
DEBUG[11-16|14:39:02.089] Pruned state history items=1 tailid=314,636
INFO [11-16|14:39:02.089] Imported new potential chain segment number=18,585,097 hash=1f5a29..cd2730 blocks=1 txs=164 mgas=20.639 elapsed=98.820ms mgasps=208.849 snapdiffs=8.40MiB triediffs=178.66MiB triedirty=51.30MiB
INFO [11-16|14:39:02.203] Chain head was updated number=18,585,097 hash=1f5a29..cd2730 root=75acd3..ebef96 elapsed=3.319101ms
DEBUG[11-16|14:39:12.917] Created new diff layer id=404,764 block=18,585,098 nodes=4684 size=1.73MiB
DEBUG[11-16|14:39:13.060] Stored state history id=404,636 block=18,584,970 data=15.44KiB index=25.74KiB elapsed=142.718ms
DEBUG[11-16|14:39:13.062] Pruned state history items=1 tailid=314,637
INFO [11-16|14:39:13.062] Imported new potential chain segment number=18,585,098 hash=c5acb8..458385 blocks=1 txs=241 mgas=17.906 elapsed=234.884ms mgasps=76.232 snapdiffs=8.45MiB triediffs=179.02MiB triedirty=51.69MiB
INFO [11-16|14:39:13.163] Chain head was updated number=18,585,098 hash=c5acb8..458385 root=952e65..f0d2c6 elapsed=2.81972ms
DEBUG[11-16|14:39:24.738] Created new diff layer id=404,765 block=18,585,099 nodes=3315 size=1.21MiB
DEBUG[11-16|14:39:26.699] Stored state history id=404,637 block=18,584,971 data=13.31KiB index=21.57KiB elapsed=1.961s
DEBUG[11-16|14:39:26.705] Pruned state history items=1 tailid=314,638
INFO [11-16|14:39:26.705] Imported new potential chain segment number=18,585,099 hash=55fd46..a25334 blocks=1 txs=136 mgas=14.114 elapsed=2.037s mgasps=6.926 snapdiffs=8.49MiB triediffs=179.08MiB triedirty=51.95MiB
INFO [11-16|14:39:26.884] Chain head was updated number=18,585,099 hash=55fd46..a25334 root=6466ff..37c521 elapsed=8.268105ms
DEBUG[11-16|14:39:36.036] Created new diff layer id=404,766 block=18,585,100 nodes=2503 size=974.99KiB
DEBUG[11-16|14:39:37.970] Stored state history id=404,638 block=18,584,972 data=19.00KiB index=35.61KiB elapsed=1.933s
DEBUG[11-16|14:39:37.972] Pruned state history items=1 tailid=314,639
INFO [11-16|14:39:37.972] Imported new potential chain segment number=18,585,100 hash=c88a05..3d2dca blocks=1 txs=134 mgas=10.422 elapsed=1.978s mgasps=5.268 snapdiffs=8.51MiB triediffs=178.70MiB triedirty=52.44MiB
INFO [11-16|14:39:38.140] Chain head was updated number=18,585,100 hash=c88a05..3d2dca root=b4a8a6..14bf4d elapsed=2.516947ms
DEBUG[11-16|14:39:49.062] Created new diff layer id=404,767 block=18,585,101 nodes=2769 size=1.04MiB
DEBUG[11-16|14:39:50.934] Stored state history id=404,639 block=18,584,973 data=17.71KiB index=27.72KiB elapsed=1.871s
DEBUG[11-16|14:39:50.936] Pruned state history items=1 tailid=314,640
INFO [11-16|14:39:50.936] Imported new potential chain segment number=18,585,101 hash=e21791..371d0c blocks=1 txs=108 mgas=10.488 elapsed=1.919s mgasps=5.463 snapdiffs=8.54MiB triediffs=178.29MiB triedirty=52.75MiB
INFO [11-16|14:39:51.051] Chain head was updated number=18,585,101 hash=e21791..371d0c root=88b7d1..55b21d elapsed=2.693409ms
DEBUG[11-16|14:40:00.669] Created new diff layer id=404,768 block=18,585,102 nodes=4281 size=1.59MiB
DEBUG[11-16|14:40:02.615] Stored state history id=404,640 block=18,584,974 data=13.98KiB index=22.48KiB elapsed=1.945s
DEBUG[11-16|14:40:02.617] Pruned state history items=1 tailid=314,641
INFO [11-16|14:40:02.617] Imported new potential chain segment number=18,585,102 hash=6d1f6a..a3321b blocks=1 txs=231 mgas=29.452 elapsed=2.100s mgasps=14.020 snapdiffs=8.58MiB triediffs=178.64MiB triedirty=53.00MiB
INFO [11-16|14:40:02.738] Chain head was updated number=18,585,102 hash=6d1f6a..a3321b root=560bb2..397dfa elapsed=5.912951ms
DEBUG[11-16|14:40:12.957] Created new diff layer id=404,769 block=18,585,103 nodes=4093 size=1.52MiB
DEBUG[11-16|14:40:14.824] Stored state history id=404,641 block=18,584,975 data=14.92KiB index=25.50KiB elapsed=1.866s
DEBUG[11-16|14:40:14.850] Pruned state history items=1 tailid=314,642
INFO [11-16|14:40:14.850] Imported new potential chain segment number=18,585,103 hash=eb1284..5cc5d3 blocks=1 txs=226 mgas=19.502 elapsed=1.981s mgasps=9.840 snapdiffs=8.63MiB triediffs=178.79MiB triedirty=53.36MiB
INFO [11-16|14:40:15.029] Chain head was updated number=18,585,103 hash=eb1284..5cc5d3 root=748466..35a34e elapsed=7.766106ms
DEBUG[11-16|14:40:26.624] Created new diff layer id=404,770 block=18,585,104 nodes=3544 size=1.31MiB
DEBUG[11-16|14:40:26.756] Stored state history id=404,642 block=18,584,976 data=13.71KiB index=22.26KiB elapsed=132.273ms
DEBUG[11-16|14:40:26.758] Pruned state history items=1 tailid=314,643
INFO [11-16|14:40:26.759] Imported new potential chain segment number=18,585,104 hash=291f63..138e46 blocks=1 txs=168 mgas=16.699 elapsed=191.655ms mgasps=87.129 snapdiffs=8.66MiB triediffs=178.89MiB triedirty=53.62MiB
INFO [11-16|14:40:28.382] Chain head was updated number=18,585,104 hash=291f63..138e46 root=b2acb1..787c54 elapsed=2.523082ms
DEBUG[11-16|14:40:37.499] Created new diff layer id=404,771 block=18,585,105 nodes=3189 size=1.16MiB
DEBUG[11-16|14:40:37.557] Stored state history id=404,643 block=18,584,977 data=14.60KiB index=24.67KiB elapsed=57.794ms
DEBUG[11-16|14:40:37.570] Pruned state history items=1 tailid=314,644
INFO [11-16|14:40:37.570] Imported new potential chain segment number=18,585,105 hash=a10edd..f0a725 blocks=1 txs=143 mgas=16.401 elapsed=198.476ms mgasps=82.635 snapdiffs=8.70MiB triediffs=178.63MiB triedirty=53.95MiB
INFO [11-16|14:40:37.739] Chain head was updated number=18,585,105 hash=a10edd..f0a725 root=b34794..2c0114 elapsed=2.930667ms
DEBUG[11-16|14:40:47.983] Created new diff layer id=404,772 block=18,585,106 nodes=3199 size=1.18MiB
DEBUG[11-16|14:40:48.007] Stored state history id=404,644 block=18,584,978 data=15.65KiB index=24.87KiB elapsed=24.268ms
DEBUG[11-16|14:40:48.009] Pruned state history items=1 tailid=314,645
INFO [11-16|14:40:48.009] Imported new potential chain segment number=18,585,106 hash=5ba1cb..029083 blocks=1 txs=163 mgas=13.600 elapsed=81.102ms mgasps=167.692 snapdiffs=8.73MiB triediffs=178.53MiB triedirty=54.23MiB
INFO [11-16|14:40:48.160] Chain head was updated number=18,585,106 hash=5ba1cb..029083 root=bcc712..28d71f elapsed=2.351197ms
DEBUG[11-16|14:41:01.243] Created new diff layer id=404,773 block=18,585,107 nodes=3380 size=1.25MiB
DEBUG[11-16|14:41:01.429] Stored state history id=404,645 block=18,584,979 data=19.39KiB index=31.10KiB elapsed=185.772ms
DEBUG[11-16|14:41:01.431] Pruned state history items=1 tailid=314,646
INFO [11-16|14:41:01.431] Imported new potential chain segment number=18,585,107 hash=01abc6..91364b blocks=1 txs=137 mgas=15.080 elapsed=240.395ms mgasps=62.728 snapdiffs=8.77MiB triediffs=178.04MiB triedirty=54.61MiB
INFO [11-16|14:41:01.553] Chain head was updated number=18,585,107 hash=01abc6..91364b root=cdbe68..cb2e50 elapsed=2.376941ms
DEBUG[11-16|14:41:12.991] Created new diff layer id=404,774 block=18,585,108 nodes=4120 size=1.54MiB
DEBUG[11-16|14:41:14.885] Stored state history id=404,646 block=18,584,980 data=15.53KiB index=23.62KiB elapsed=1.894s
DEBUG[11-16|14:41:14.891] Pruned state history items=1 tailid=314,647
INFO [11-16|14:41:14.891] Imported new potential chain segment number=18,585,108 hash=301a2e..2975d8 blocks=1 txs=207 mgas=14.644 elapsed=1.971s mgasps=7.427 snapdiffs=8.81MiB triediffs=178.26MiB triedirty=54.88MiB
INFO [11-16|14:41:15.140] Chain head was updated number=18,585,108 hash=301a2e..2975d8 root=852277..fd5063 elapsed=3.46671ms
DEBUG[11-16|14:41:25.855] Created new diff layer id=404,775 block=18,585,109 nodes=3702 size=1.36MiB
DEBUG[11-16|14:41:27.805] Stored state history id=404,647 block=18,584,981 data=14.12KiB index=24.24KiB elapsed=1.949s
DEBUG[11-16|14:41:27.807] Pruned state history items=1 tailid=314,648
INFO [11-16|14:41:27.808] Imported new potential chain segment number=18,585,109 hash=ce9351..f5b974 blocks=1 txs=165 mgas=16.114 elapsed=2.015s mgasps=7.993 snapdiffs=8.85MiB triediffs=178.29MiB triedirty=55.16MiB
INFO [11-16|14:41:27.969] Chain head was updated number=18,585,109 hash=ce9351..f5b974 root=2d414d..151666 elapsed=11.646584ms
DEBUG[11-16|14:41:37.592] Created new diff layer id=404,776 block=18,585,110 nodes=3130 size=1.17MiB
DEBUG[11-16|14:41:39.479] Stored state history id=404,648 block=18,584,982 data=16.59KiB index=27.75KiB elapsed=1.887s
DEBUG[11-16|14:41:39.482] Pruned state history items=1 tailid=314,649
INFO [11-16|14:41:39.482] Imported new potential chain segment number=18,585,110 hash=68b9bb..4b8899 blocks=1 txs=141 mgas=20.500 elapsed=1.971s mgasps=10.396 snapdiffs=8.88MiB triediffs=177.97MiB triedirty=55.51MiB
INFO [11-16|14:41:39.632] Chain head was updated number=18,585,110 hash=68b9bb..4b8899 root=266519..300ec2 elapsed=7.314154ms
DEBUG[11-16|14:41:48.849] Created new diff layer id=404,777 block=18,585,111 nodes=2860 size=1.07MiB
DEBUG[11-16|14:41:50.737] Stored state history id=404,649 block=18,584,983 data=15.18KiB index=22.44KiB elapsed=1.887s
DEBUG[11-16|14:41:50.739] Pruned state history items=1 tailid=314,650
INFO [11-16|14:41:50.739] Imported new potential chain segment number=18,585,111 hash=36db1a..5b6dcc blocks=1 txs=151 mgas=14.844 elapsed=1.938s mgasps=7.658 snapdiffs=8.91MiB triediffs=177.83MiB triedirty=55.75MiB
INFO [11-16|14:41:50.889] Chain head was updated number=18,585,111 hash=36db1a..5b6dcc root=4b83d6..0a5b00 elapsed=4.204821ms
DEBUG[11-16|14:42:01.386] Created new diff layer id=404,778 block=18,585,112 nodes=2901 size=1.08MiB
DEBUG[11-16|14:42:03.443] Stored state history id=404,650 block=18,584,984 data=18.52KiB index=30.96KiB elapsed=2.056s
DEBUG[11-16|14:42:03.464] Pruned state history items=1 tailid=314,651
INFO [11-16|14:42:03.464] Imported new potential chain segment number=18,585,112 hash=20bd42..bf8597 blocks=1 txs=126 mgas=12.495 elapsed=2.265s mgasps=5.516 snapdiffs=4.91MiB triediffs=177.17MiB triedirty=56.13MiB
INFO [11-16|14:42:03.672] Chain head was updated number=18,585,112 hash=20bd42..bf8597 root=0ca33b..fdaff1 elapsed=4.683875ms
DEBUG[11-16|14:42:13.504] Created new diff layer id=404,779 block=18,585,113 nodes=3314 size=1.24MiB
DEBUG[11-16|14:42:14.094] Stored state history id=404,651 block=18,584,985 data=17.58KiB index=31.85KiB elapsed=590.288ms
DEBUG[11-16|14:42:14.096] Pruned state history items=1 tailid=314,652
INFO [11-16|14:42:14.096] Imported new potential chain segment number=18,585,113 hash=214796..9664c4 blocks=1 txs=173 mgas=13.488 elapsed=653.682ms mgasps=20.633 snapdiffs=4.94MiB triediffs=176.79MiB triedirty=56.56MiB
INFO [11-16|14:42:14.246] Chain head was updated number=18,585,113 hash=214796..9664c4 root=6c87d1..232566 elapsed=3.489952ms
DEBUG[11-16|14:42:24.356] Created new diff layer id=404,780 block=18,585,114 nodes=3553 size=1.30MiB
DEBUG[11-16|14:42:24.473] Stored state history id=404,652 block=18,584,986 data=8.63KiB index=15.88KiB elapsed=116.830ms
DEBUG[11-16|14:42:24.474] Pruned state history items=1 tailid=314,653
INFO [11-16|14:42:24.474] Imported new potential chain segment number=18,585,114 hash=f09347..71f3a6 blocks=1 txs=144 mgas=14.589 elapsed=179.805ms mgasps=81.137 snapdiffs=4.97MiB triediffs=177.15MiB triedirty=56.79MiB
INFO [11-16|14:42:24.600] Chain head was updated number=18,585,114 hash=f09347..71f3a6 root=6897e9..66d406 elapsed=2.427651ms
DEBUG[11-16|14:42:37.254] Created new diff layer id=404,781 block=18,585,115 nodes=3396 size=1.27MiB
DEBUG[11-16|14:42:37.282] Stored state history id=404,653 block=18,584,987 data=19.22KiB index=31.24KiB elapsed=28.397ms
DEBUG[11-16|14:42:37.285] Pruned state history items=1 tailid=314,654
INFO [11-16|14:42:37.285] Imported new potential chain segment number=18,585,115 hash=976371..f23cbe blocks=1 txs=162 mgas=14.600 elapsed=84.886ms mgasps=171.993 snapdiffs=5.01MiB triediffs=176.79MiB triedirty=57.19MiB
INFO [11-16|14:42:37.408] Chain head was updated number=18,585,115 hash=976371..f23cbe root=adea53..c5b06d elapsed=2.54273ms
DEBUG[11-16|14:42:48.353] Created new diff layer id=404,782 block=18,585,116 nodes=3108 size=1.13MiB
DEBUG[11-16|14:42:48.437] Stored state history id=404,654 block=18,584,988 data=12.97KiB index=21.06KiB elapsed=84.746ms
DEBUG[11-16|14:42:48.440] Pruned state history items=1 tailid=314,655
INFO [11-16|14:42:48.441] Imported new potential chain segment number=18,585,116 hash=f563df..efeaf8 blocks=1 txs=129 mgas=16.850 elapsed=140.173ms mgasps=120.210 snapdiffs=5.69MiB triediffs=176.73MiB triedirty=57.43MiB
INFO [11-16|14:42:48.566] Chain head was updated number=18,585,116 hash=f563df..efeaf8 root=9426e7..c9189d elapsed=2.179669ms
DEBUG[11-16|14:43:01.303] Created new diff layer id=404,783 block=18,585,117 nodes=3446 size=1.25MiB
DEBUG[11-16|14:43:01.663] Stored state history id=404,655 block=18,584,989 data=16.44KiB index=29.06KiB elapsed=360.098ms
DEBUG[11-16|14:43:01.665] Pruned state history items=1 tailid=314,656
INFO [11-16|14:43:01.665] Imported new potential chain segment number=18,585,117 hash=10eff9..c6d210 blocks=1 txs=166 mgas=23.522 elapsed=437.622ms mgasps=53.750 snapdiffs=5.72MiB triediffs=176.37MiB triedirty=57.77MiB
INFO [11-16|14:43:01.808] Chain head was updated number=18,585,117 hash=10eff9..c6d210 root=e19541..476129 elapsed=3.190976ms
DEBUG[11-16|14:43:12.917] Created new diff layer id=404,784 block=18,585,118 nodes=3706 size=1.39MiB
DEBUG[11-16|14:43:14.967] Stored state history id=404,656 block=18,584,990 data=11.56KiB index=19.20KiB elapsed=2.050s
DEBUG[11-16|14:43:14.969] Pruned state history items=1 tailid=314,657
INFO [11-16|14:43:14.969] Imported new potential chain segment number=18,585,118 hash=07a202..58df4e blocks=1 txs=181 mgas=19.341 elapsed=2.110s mgasps=9.164 snapdiffs=5.76MiB triediffs=176.65MiB triedirty=58.02MiB
INFO [11-16|14:43:15.120] Chain head was updated number=18,585,118 hash=07a202..58df4e root=c7cc64..1e2f3f elapsed=4.140722ms
DEBUG[11-16|14:43:24.827] Created new diff layer id=404,785 block=18,585,119 nodes=2236 size=868.43KiB
DEBUG[11-16|14:43:26.784] Stored state history id=404,657 block=18,584,991 data=14.54KiB index=25.42KiB elapsed=1.955s
DEBUG[11-16|14:43:26.786] Pruned state history items=1 tailid=314,658
INFO [11-16|14:43:26.786] Imported new potential chain segment number=18,585,119 hash=c198b0..6232af blocks=1 txs=99 mgas=8.867 elapsed=1.990s mgasps=4.455 snapdiffs=5.78MiB triediffs=176.17MiB triedirty=58.31MiB
INFO [11-16|14:43:26.922] Chain head was updated number=18,585,119 hash=c198b0..6232af root=5860ad..5af431 elapsed=1.981379ms
DEBUG[11-16|14:43:36.364] Created new diff layer id=404,786 block=18,585,120 nodes=3582 size=1.32MiB
DEBUG[11-16|14:43:38.250] Stored state history id=404,658 block=18,584,992 data=16.49KiB index=27.14KiB elapsed=1.885s
DEBUG[11-16|14:43:38.252] Pruned state history items=1 tailid=314,659
INFO [11-16|14:43:38.252] Imported new potential chain segment number=18,585,120 hash=aea5d8..1aa329 blocks=1 txs=148 mgas=15.964 elapsed=1.966s mgasps=8.117 snapdiffs=5.82MiB triediffs=176.05MiB triedirty=58.65MiB
INFO [11-16|14:43:38.394] Chain head was updated number=18,585,120 hash=aea5d8..1aa329 root=c70142..e81a71 elapsed=3.127554ms
DEBUG[11-16|14:43:48.793] Created new diff layer id=404,787 block=18,585,121 nodes=2486 size=950.85KiB
DEBUG[11-16|14:43:50.694] Stored state history id=404,659 block=18,584,993 data=16.74KiB index=27.25KiB elapsed=1.900s
DEBUG[11-16|14:43:50.696] Pruned state history items=1 tailid=314,660
INFO [11-16|14:43:50.696] Imported new potential chain segment number=18,585,121 hash=6cc90e..cd13ca blocks=1 txs=117 mgas=8.845 elapsed=1.942s mgasps=4.554 snapdiffs=5.85MiB triediffs=175.57MiB triedirty=58.97MiB
INFO [11-16|14:43:50.833] Chain head was updated number=18,585,121 hash=6cc90e..cd13ca root=c67109..320dae elapsed=2.163996ms
DEBUG[11-16|14:44:01.843] Created new diff layer id=404,788 block=18,585,122 nodes=3698 size=1.33MiB
DEBUG[11-16|14:44:03.919] Stored state history id=404,660 block=18,584,994 data=15.25KiB index=27.11KiB elapsed=2.076s
DEBUG[11-16|14:44:03.925] Pruned state history items=1 tailid=314,661
INFO [11-16|14:44:03.926] Imported new potential chain segment number=18,585,122 hash=dd7e36..1fe896 blocks=1 txs=134 mgas=16.565 elapsed=2.193s mgasps=7.552 snapdiffs=5.89MiB triediffs=175.42MiB triedirty=59.37MiB
INFO [11-16|14:44:04.064] Chain head was updated number=18,585,122 hash=dd7e36..1fe896 root=974133..0c17a3 elapsed=3.090148ms
DEBUG[11-16|14:44:14.194] Created new diff layer id=404,789 block=18,585,123 nodes=3526 size=1.33MiB
DEBUG[11-16|14:44:14.883] Stored state history id=404,661 block=18,584,995 data=12.21KiB index=21.18KiB elapsed=688.503ms
DEBUG[11-16|14:44:14.885] Pruned state history items=1 tailid=314,662
INFO [11-16|14:44:14.885] Imported new potential chain segment number=18,585,123 hash=779abc..f5f563 blocks=1 txs=160 mgas=13.027 elapsed=834.321ms mgasps=15.614 snapdiffs=5.92MiB triediffs=175.61MiB triedirty=59.62MiB
INFO [11-16|14:44:15.036] Chain head was updated number=18,585,123 hash=779abc..f5f563 root=8ab97c..851a49 elapsed=2.754219ms
DEBUG[11-16|14:44:23.836] Created new diff layer id=404,790 block=18,585,124 nodes=2344 size=889.03KiB
DEBUG[11-16|14:44:23.957] Stored state history id=404,662 block=18,584,996 data=18.56KiB index=32.34KiB elapsed=120.811ms
DEBUG[11-16|14:44:23.960] Pruned state history items=1 tailid=314,663
INFO [11-16|14:44:23.960] Imported new potential chain segment number=18,585,124 hash=2a26e2..de51a3 blocks=1 txs=112 mgas=8.253 elapsed=160.487ms mgasps=51.426 snapdiffs=5.95MiB triediffs=174.71MiB triedirty=60.05MiB
INFO [11-16|14:44:24.074] Chain head was updated number=18,585,124 hash=2a26e2..de51a3 root=5993cc..b773ea elapsed=1.459977ms
DEBUG[11-16|14:44:36.369] Created new diff layer id=404,791 block=18,585,125 nodes=3880 size=1.45MiB
DEBUG[11-16|14:44:36.398] Stored state history id=404,663 block=18,584,997 data=7.78KiB index=13.65KiB elapsed=28.220ms
DEBUG[11-16|14:44:36.399] Pruned state history items=1 tailid=314,664
INFO [11-16|14:44:36.399] Imported new potential chain segment number=18,585,125 hash=a343a6..19b8e5 blocks=1 txs=200 mgas=16.582 elapsed=95.710ms mgasps=173.247 snapdiffs=6.16MiB triediffs=175.39MiB triedirty=60.22MiB
INFO [11-16|14:44:36.541] Chain head was updated number=18,585,125 hash=a343a6..19b8e5 root=4f525a..11ee1e elapsed=2.915446ms
DEBUG[11-16|14:44:48.528] Created new diff layer id=404,792 block=18,585,126 nodes=2516 size=974.66KiB
DEBUG[11-16|14:44:48.624] Stored state history id=404,664 block=18,584,998 data=20.67KiB index=34.99KiB elapsed=96.315ms
DEBUG[11-16|14:44:48.627] Pruned state history items=1 tailid=314,665
INFO [11-16|14:44:48.627] Imported new potential chain segment number=18,585,126 hash=84db2c..1b1042 blocks=1 txs=113 mgas=11.715 elapsed=135.980ms mgasps=86.151 snapdiffs=6.19MiB triediffs=174.62MiB triedirty=60.62MiB
INFO [11-16|14:44:48.742] Chain head was updated number=18,585,126 hash=84db2c..1b1042 root=4c1fce..12341d elapsed=2.224823ms
DEBUG[11-16|14:45:00.536] Created new diff layer id=404,793 block=18,585,127 nodes=3299 size=1.24MiB
DEBUG[11-16|14:45:01.245] Stored state history id=404,665 block=18,584,999 data=17.76KiB index=30.33KiB elapsed=708.945ms
DEBUG[11-16|14:45:01.248] Pruned state history items=1 tailid=314,666
INFO [11-16|14:45:01.248] Imported new potential chain segment number=18,585,127 hash=e4d623..d92605 blocks=1 txs=144 mgas=12.710 elapsed=759.609ms mgasps=16.732 snapdiffs=6.22MiB triediffs=174.19MiB triedirty=60.99MiB
INFO [11-16|14:45:01.370] Chain head was updated number=18,585,127 hash=e4d623..d92605 root=b397d2..5c4d5b elapsed=2.222996ms
DEBUG[11-16|14:45:17.708] Created new diff layer id=404,794 block=18,585,128 nodes=4448 size=1.63MiB
DEBUG[11-16|14:45:19.674] Stored state history id=404,666 block=18,585,000 data=11.68KiB index=19.55KiB elapsed=1.966s
DEBUG[11-16|14:45:19.677] Pruned state history items=1 tailid=314,667
INFO [11-16|14:45:19.677] Imported new potential chain segment number=18,585,128 hash=c739a6..d59b1f blocks=1 txs=207 mgas=25.728 elapsed=3.892s mgasps=6.610 snapdiffs=6.27MiB triediffs=174.68MiB triedirty=61.28MiB
INFO [11-16|14:45:19.793] Chain head was updated number=18,585,128 hash=c739a6..d59b1f root=a2d391..d56178 elapsed=3.735864ms
DEBUG[11-16|14:45:26.029] Created new diff layer id=404,795 block=18,585,129 nodes=3260 size=1.21MiB
The time for creating state history varies a lot regardless the size which is indeed weird and have the gut feeling that is relevant with hardware somehow.
Do you have any suggestions on what I can try next, to help debug this?
I'm leaning towards moving the ancient directory onto the NVMe drive, to remove the SSD from the equation entirely. Does that sound like a reasonable next step?
edit: I'm doing this, will report back.
After moving the ancient directory over to my main NVMe drive, the issues I was experiencing in this bug report are no longer occurring.
It remains unknown why the code changes https://github.com/ethereum/go-ethereum/compare/v1.13.4...v1.13.5 resulted in the behavior described in this issue, with my specific setup: ancient directory on a Samsung 860 EVO drive.
How do you connect the Samsung 860 EVO drive with your machine btw? Is it via the USB or?
I also encounter this issue. If ancient directory is on external HDD drive, elapsed time is on order of seconds. Even after I moved it to internal NVME drive, the elapsed time will still occasionally went up to 1s, and the average time is worse than before.
INFO [11-18|11:20:24.830] Imported new potential chain segment number=18,598,387 hash=92cc11..f04731 blocks=1 txs=165 mgas=20.980 elapsed=270.196ms mgasps=77.648 snapdiffs=5.31MiB triediffs=151.46MiB triedirty=63.75MiB
INFO [11-18|11:20:37.709] Imported new potential chain segment number=18,598,388 hash=966db1..2626e0 blocks=1 txs=148 mgas=19.310 elapsed=204.736ms mgasps=94.318 snapdiffs=5.34MiB triediffs=151.24MiB triedirty=63.99MiB
INFO [11-18|11:20:51.559] Imported new potential chain segment number=18,598,389 hash=69e903..0dd475 blocks=1 txs=217 mgas=17.244 elapsed=1.022s mgasps=16.867 snapdiffs=5.38MiB triediffs=152.32MiB triedirty=0.00B
INFO [11-18|11:21:01.566] Imported new potential chain segment number=18,598,390 hash=ff38a4..a5e39e blocks=1 txs=120 mgas=16.078 elapsed=229.834ms mgasps=69.953 snapdiffs=5.40MiB triediffs=151.31MiB triedirty=1.75MiB
INFO [11-18|11:21:13.267] Imported new potential chain segment number=18,598,391 hash=594765..c7186e blocks=1 txs=119 mgas=16.884 elapsed=155.748ms mgasps=108.406 snapdiffs=5.43MiB triediffs=150.95MiB triedirty=2.57MiB
INFO [11-18|11:21:24.856] Imported new potential chain segment number=18,598,392 hash=95db5c..dd826d blocks=1 txs=123 mgas=10.379 elapsed=117.302ms mgasps=88.481 snapdiffs=5.46MiB triediffs=151.13MiB triedirty=3.07MiB
INFO [11-18|11:21:38.800] Imported new potential chain segment number=18,598,393 hash=81eda9..c7c3ba blocks=1 txs=190 mgas=16.984 elapsed=166.668ms mgasps=101.906 snapdiffs=5.49MiB triediffs=150.97MiB triedirty=3.86MiB
INFO [11-18|11:21:49.904] Imported new potential chain segment number=18,598,394 hash=948658..106003 blocks=1 txs=120 mgas=11.771 elapsed=123.688ms mgasps=95.167 snapdiffs=5.51MiB triediffs=150.69MiB triedirty=4.47MiB
INFO [11-18|11:22:02.355] Imported new potential chain segment number=18,598,395 hash=34a8d3..533745 blocks=1 txs=154 mgas=18.580 elapsed=142.287ms mgasps=130.581 snapdiffs=5.55MiB triediffs=150.92MiB triedirty=4.96MiB
INFO [11-18|11:22:12.617] Imported new potential chain segment number=18,598,396 hash=fdc97a..6629d3 blocks=1 txs=177 mgas=10.035 elapsed=101.211ms mgasps=99.150 snapdiffs=5.58MiB triediffs=151.07MiB triedirty=5.39MiB
INFO [11-18|11:22:27.755] Imported new potential chain segment number=18,598,397 hash=29e2af..5db52f blocks=1 txs=204 mgas=10.467 elapsed=254.070ms mgasps=41.198 snapdiffs=5.60MiB triediffs=150.42MiB triedirty=6.16MiB
INFO [11-18|11:22:37.474] Imported new potential chain segment number=18,598,398 hash=283913..669646 blocks=1 txs=222 mgas=15.339 elapsed=332.487ms mgasps=46.133 snapdiffs=5.64MiB triediffs=150.66MiB triedirty=6.69MiB
INFO [11-18|11:22:49.822] Imported new potential chain segment number=18,598,399 hash=c1fe0d..71b197 blocks=1 txs=135 mgas=12.025 elapsed=513.577ms mgasps=23.414 snapdiffs=5.67MiB triediffs=150.73MiB triedirty=7.26MiB
INFO [11-18|11:23:02.874] Imported new potential chain segment number=18,598,400 hash=f3552c..90e0fd blocks=1 txs=176 mgas=17.793 elapsed=838.678ms mgasps=21.215 snapdiffs=5.71MiB triediffs=150.84MiB triedirty=7.80MiB
INFO [11-18|11:23:13.204] Imported new potential chain segment number=18,598,401 hash=ed6911..9fdaa9 blocks=1 txs=178 mgas=13.868 elapsed=380.819ms mgasps=36.416 snapdiffs=5.74MiB triediffs=150.83MiB triedirty=8.27MiB
INFO [11-18|11:23:25.643] Imported new potential chain segment number=18,598,402 hash=391550..b9946d blocks=1 txs=187 mgas=15.377 elapsed=564.806ms mgasps=27.225 snapdiffs=5.77MiB triediffs=150.89MiB triedirty=8.78MiB
INFO [11-18|11:23:37.320] Imported new potential chain segment number=18,598,403 hash=9849bf..907bc4 blocks=1 txs=153 mgas=12.929 elapsed=380.154ms mgasps=34.011 snapdiffs=5.80MiB triediffs=150.68MiB triedirty=9.41MiB
INFO [11-18|11:23:50.059] Imported new potential chain segment number=18,598,404 hash=002cd9..e84145 blocks=1 txs=151 mgas=11.585 elapsed=111.563ms mgasps=103.841 snapdiffs=5.83MiB triediffs=150.70MiB triedirty=9.84MiB
INFO [11-18|11:24:01.743] Imported new potential chain segment number=18,598,405 hash=435295..56f7d5 blocks=1 txs=243 mgas=19.523 elapsed=160.237ms mgasps=121.837 snapdiffs=5.87MiB triediffs=151.15MiB triedirty=10.26MiB
INFO [11-18|11:24:15.125] Imported new potential chain segment number=18,598,406 hash=99a77c..f2bae1 blocks=1 txs=164 mgas=14.849 elapsed=117.826ms mgasps=126.025 snapdiffs=5.90MiB triediffs=150.96MiB triedirty=10.84MiB
INFO [11-18|11:24:25.088] Imported new potential chain segment number=18,598,407 hash=aae474..90723a blocks=1 txs=156 mgas=15.495 elapsed=148.611ms mgasps=104.266 snapdiffs=5.94MiB triediffs=150.94MiB triedirty=11.39MiB
INFO [11-18|11:24:37.423] Imported new potential chain segment number=18,598,408 hash=0992e1..34f78f blocks=1 txs=175 mgas=14.622 elapsed=156.366ms mgasps=93.512 snapdiffs=5.97MiB triediffs=150.36MiB triedirty=12.13MiB
INFO [11-18|11:24:48.443] Imported new potential chain segment number=18,598,409 hash=be9424..865bf4 blocks=1 txs=146 mgas=11.285 elapsed=102.940ms mgasps=109.631 snapdiffs=5.99MiB triediffs=150.13MiB triedirty=12.56MiB
INFO [11-18|11:25:00.812] Imported new potential chain segment number=18,598,410 hash=bb54ba..23ebb2 blocks=1 txs=178 mgas=16.393 elapsed=158.266ms mgasps=103.580 snapdiffs=6.03MiB triediffs=150.30MiB triedirty=13.07MiB
INFO [11-18|11:25:12.963] Imported new potential chain segment number=18,598,411 hash=de04ef..210038 blocks=1 txs=185 mgas=11.464 elapsed=136.719ms mgasps=83.850 snapdiffs=6.07MiB triediffs=150.59MiB triedirty=13.39MiB
INFO [11-18|11:25:25.238] Imported new potential chain segment number=18,598,412 hash=6fec61..e56331 blocks=1 txs=160 mgas=9.642 elapsed=96.816ms mgasps=99.593 snapdiffs=6.09MiB triediffs=150.25MiB triedirty=13.91MiB
INFO [11-18|11:25:38.405] Imported new potential chain segment number=18,598,413 hash=b38608..f089c6 blocks=1 txs=378 mgas=28.648 elapsed=908.621ms mgasps=31.529 snapdiffs=6.16MiB triediffs=151.89MiB triedirty=14.25MiB
INFO [11-18|11:25:49.143] Imported new potential chain segment number=18,598,414 hash=7d60b2..c6b8dd blocks=1 txs=153 mgas=11.539 elapsed=104.758ms mgasps=110.148 snapdiffs=6.19MiB triediffs=151.74MiB triedirty=14.70MiB
INFO [11-18|11:26:02.922] Imported new potential chain segment number=18,598,415 hash=5615db..1c3c9b blocks=1 txs=151 mgas=11.881 elapsed=140.968ms mgasps=84.283 snapdiffs=6.21MiB triediffs=150.91MiB triedirty=15.46MiB
INFO [11-18|11:26:12.622] Imported new potential chain segment number=18,598,416 hash=f54be2..d32c1c blocks=1 txs=149 mgas=14.689 elapsed=136.151ms mgasps=107.884 snapdiffs=6.25MiB triediffs=151.26MiB triedirty=15.81MiB
INFO [11-18|11:26:25.324] Imported new potential chain segment number=18,598,417 hash=a4d895..f01146 blocks=1 txs=146 mgas=16.186 elapsed=158.412ms mgasps=102.175 snapdiffs=6.28MiB triediffs=150.60MiB triedirty=16.49MiB
INFO [11-18|11:26:36.516] Imported new potential chain segment number=18,598,418 hash=3a9c6b..d92264 blocks=1 txs=106 mgas=8.665 elapsed=75.308ms mgasps=115.065 snapdiffs=6.30MiB triediffs=149.88MiB triedirty=16.99MiB
INFO [11-18|11:26:48.966] Imported new potential chain segment number=18,598,419 hash=9cd2b4..45321b blocks=1 txs=224 mgas=25.593 elapsed=188.948ms mgasps=135.448 snapdiffs=6.34MiB triediffs=150.44MiB triedirty=17.38MiB
INFO [11-18|11:20:51.559] Imported new potential chain segment number=18,598,389 hash=69e903..0dd475 blocks=1 txs=217 mgas=17.244 elapsed=1.022s mgasps=16.867 snapdiffs=5.38MiB triediffs=152.32MiB triedirty=0.00B
The reason for 1s time execution is in path based scheme, Geth needs to flush the accumulated state changes into database in a single atomic batch, and it takes 1s time.
We will try to test the performance with external HDD setup. If possible, please also share some logs with external HDD.
We will try to test the performance with external HDD setup. If possible, please also share some logs with external HDD.
Below is the log with an internal HDD setup
INFO [11-19|23:35:39.099] Imported new potential chain segment number=18,609,163 hash=fd345c..f2df39 blocks=1 txs=160 mgas=14.962 elapsed=1.575s mgasps=9.497 snapdiffs=5.17MiB triediffs=170.78MiB triedirty=41.08MiB
INFO [11-19|23:35:39.504] Chain head was updated number=18,609,163 hash=fd345c..f2df39 root=61c216..4e880a elapsed=7.097546ms
INFO [11-19|23:35:53.276] Imported new potential chain segment number=18,609,164 hash=89a97d..ffed65 blocks=1 txs=135 mgas=14.611 elapsed=3.092s mgasps=4.724 snapdiffs=5.20MiB triediffs=170.59MiB triedirty=41.53MiB
INFO [11-19|23:35:53.678] Chain head was updated number=18,609,164 hash=89a97d..ffed65 root=6ca3c0..d12da3 elapsed=8.309445ms
INFO [11-19|23:36:02.798] Imported new potential chain segment number=18,609,165 hash=3b3ce6..beefa2 blocks=1 txs=115 mgas=13.983 elapsed=1.581s mgasps=8.842 snapdiffs=5.24MiB triediffs=170.03MiB triedirty=42.00MiB
INFO [11-19|23:36:03.103] Chain head was updated number=18,609,165 hash=3b3ce6..beefa2 root=8c69f7..a649a5 elapsed=7.229521ms
INFO [11-19|23:36:15.988] Imported new potential chain segment number=18,609,166 hash=57c080..a384d1 blocks=1 txs=161 mgas=13.704 elapsed=1.567s mgasps=8.742 snapdiffs=5.28MiB triediffs=170.28MiB triedirty=42.27MiB
INFO [11-19|23:36:16.229] Chain head was updated number=18,609,166 hash=57c080..a384d1 root=31e1f6..1ff452 elapsed=8.682012ms
INFO [11-19|23:36:27.920] Imported new potential chain segment number=18,609,167 hash=d259a0..3013a3 blocks=1 txs=21 mgas=4.428 elapsed=1.583s mgasps=2.796 snapdiffs=5.29MiB triediffs=169.56MiB triedirty=42.51MiB
INFO [11-19|23:36:28.155] Chain head was updated number=18,609,167 hash=d259a0..3013a3 root=42a07c..ae1205 elapsed=3.513604ms
INFO [11-19|23:36:39.484] Imported new potential chain segment number=18,609,168 hash=f9b362..64d1b3 blocks=1 txs=255 mgas=29.998 elapsed=1.813s mgasps=16.542 snapdiffs=5.36MiB triediffs=170.71MiB triedirty=42.74MiB
INFO [11-19|23:36:39.746] Chain head was updated number=18,609,168 hash=f9b362..64d1b3 root=7702f1..239de8 elapsed=11.396413ms
INFO [11-19|23:36:51.036] Imported new potential chain segment number=18,609,169 hash=7c8678..7d2363 blocks=1 txs=222 mgas=29.965 elapsed=2.076s mgasps=14.427 snapdiffs=5.42MiB triediffs=171.60MiB triedirty=43.05MiB
INFO [11-19|23:36:52.081] Chain head was updated number=18,609,169 hash=7c8678..7d2363 root=f5db14..4a3ac0 elapsed=25.129077ms
INFO [11-19|23:37:03.200] Imported new potential chain segment number=18,609,170 hash=d215ac..0db922 blocks=1 txs=130 mgas=18.285 elapsed=1.693s mgasps=10.799 snapdiffs=5.46MiB triediffs=171.79MiB triedirty=43.36MiB
INFO [11-19|23:37:03.467] Chain head was updated number=18,609,170 hash=d215ac..0db922 root=79a107..d6093a elapsed=8.957033ms
INFO [11-19|23:37:14.522] Imported new potential chain segment number=18,609,171 hash=83eca1..ac7ed7 blocks=1 txs=152 mgas=15.653 elapsed=1.627s mgasps=9.620 snapdiffs=5.51MiB triediffs=171.70MiB triedirty=43.82MiB
INFO [11-19|23:37:14.769] Chain head was updated number=18,609,171 hash=83eca1..ac7ed7 root=82ee85..100e5d elapsed=6.930972ms
INFO [11-19|23:37:27.151] Imported new potential chain segment number=18,609,172 hash=757292..c8735d blocks=1 txs=94 mgas=9.828 elapsed=1.742s mgasps=5.639 snapdiffs=5.53MiB triediffs=171.48MiB triedirty=44.04MiB
INFO [11-19|23:37:27.397] Chain head was updated number=18,609,172 hash=757292..c8735d root=5bc10e..99ab97 elapsed=5.982698ms
INFO [11-19|23:37:38.528] Imported new potential chain segment number=18,609,173 hash=818e3a..1a485c blocks=1 txs=97 mgas=9.780 elapsed=1.674s mgasps=5.841 snapdiffs=5.56MiB triediffs=171.01MiB triedirty=44.36MiB
INFO [11-19|23:37:38.836] Chain head was updated number=18,609,173 hash=818e3a..1a485c root=0449f3..2c1be9 elapsed=23.843359ms
INFO [11-19|23:37:50.949] Imported new potential chain segment number=18,609,174 hash=061e05..80b592 blocks=1 txs=176 mgas=18.144 elapsed=2.038s mgasps=8.900 snapdiffs=5.60MiB triediffs=170.86MiB triedirty=44.72MiB
INFO [11-19|23:37:52.078] Chain head was updated number=18,609,174 hash=061e05..80b592 root=918986..f1c783 elapsed=8.307601ms
INFO [11-19|23:38:03.254] Imported new potential chain segment number=18,609,175 hash=52ca61..bf6003 blocks=1 txs=182 mgas=18.615 elapsed=1.667s mgasps=11.166 snapdiffs=5.64MiB triediffs=170.91MiB triedirty=45.35MiB
INFO [11-19|23:38:06.444] Chain head was updated number=18,609,175 hash=52ca61..bf6003 root=4a5ceb..7d5344 elapsed=8.209521ms
INFO [11-19|23:38:13.717] Imported new potential chain segment number=18,609,176 hash=1cfe5c..216fcb blocks=1 txs=125 mgas=11.886 elapsed=1.719s mgasps=6.913 snapdiffs=5.68MiB triediffs=170.71MiB triedirty=45.66MiB
INFO [11-19|23:38:14.009] Chain head was updated number=18,609,176 hash=1cfe5c..216fcb root=b968b4..3c721a elapsed=7.134001ms
INFO [11-19|23:38:26.151] Imported new potential chain segment number=18,609,177 hash=756f16..cbb1eb blocks=1 txs=138 mgas=18.124 elapsed=1.682s mgasps=10.770 snapdiffs=5.72MiB triediffs=171.59MiB triedirty=45.80MiB
INFO [11-19|23:38:26.439] Chain head was updated number=18,609,177 hash=756f16..cbb1eb root=ddb604..48b98f elapsed=21.336292ms
INFO [11-19|23:38:38.389] Imported new potential chain segment number=18,609,178 hash=e6665a..35457e blocks=1 txs=122 mgas=11.136 elapsed=1.712s mgasps=6.504 snapdiffs=5.75MiB triediffs=170.81MiB triedirty=46.26MiB
INFO [11-19|23:38:38.676] Chain head was updated number=18,609,178 hash=e6665a..35457e root=e6034c..a7f7be elapsed=6.284588ms
INFO [11-19|23:38:50.188] Imported new potential chain segment number=18,609,179 hash=8d976e..cfecb4 blocks=1 txs=137 mgas=17.596 elapsed=1.703s mgasps=10.330 snapdiffs=5.78MiB triediffs=170.58MiB triedirty=46.64MiB
INFO [11-19|23:38:51.777] Chain head was updated number=18,609,179 hash=8d976e..cfecb4 root=f44291..7f694f elapsed=16.904469ms
INFO [11-19|23:39:02.839] Imported new potential chain segment number=18,609,180 hash=f9069d..8b7810 blocks=1 txs=112 mgas=17.053 elapsed=1.562s mgasps=10.913 snapdiffs=5.81MiB triediffs=170.40MiB triedirty=46.89MiB
INFO [11-19|23:39:03.158] Chain head was updated number=18,609,180 hash=f9069d..8b7810 root=969442..088ba3 elapsed=23.844553ms
INFO [11-19|23:39:14.904] Imported new potential chain segment number=18,609,181 hash=8b0e1f..7acd7b blocks=1 txs=155 mgas=15.812 elapsed=1.768s mgasps=8.939 snapdiffs=5.85MiB triediffs=170.28MiB triedirty=47.21MiB
INFO [11-19|23:39:15.203] Chain head was updated number=18,609,181 hash=8b0e1f..7acd7b root=094ea4..9f0d49 elapsed=7.640797ms
INFO [11-19|23:39:26.207] Imported new potential chain segment number=18,609,182 hash=80d0a7..6ba00b blocks=1 txs=126 mgas=14.183 elapsed=1.552s mgasps=9.137 snapdiffs=5.89MiB triediffs=170.30MiB triedirty=47.48MiB
INFO [11-19|23:39:26.494] Chain head was updated number=18,609,182 hash=80d0a7..6ba00b root=3bd98c..87d128 elapsed=5.980672ms
INFO [11-19|23:39:40.226] Imported new potential chain segment number=18,609,183 hash=24f850..bcde2e blocks=1 txs=149 mgas=17.266 elapsed=1.603s mgasps=10.769 snapdiffs=5.94MiB triediffs=170.48MiB triedirty=47.78MiB
INFO [11-19|23:39:40.548] Chain head was updated number=18,609,183 hash=24f850..bcde2e root=f5e6d7..ebdd24 elapsed=11.02794ms
INFO [11-19|23:39:53.378] Imported new potential chain segment number=18,609,184 hash=f819c7..969653 blocks=1 txs=101 mgas=10.731 elapsed=2.886s mgasps=3.718 snapdiffs=5.97MiB triediffs=170.13MiB triedirty=48.11MiB
INFO [11-19|23:39:53.661] Chain head was updated number=18,609,184 hash=f819c7..969653 root=7b2af4..b750bf elapsed=12.905274ms
INFO [11-19|23:40:02.437] Imported new potential chain segment number=18,609,185 hash=b2eefb..45d69d blocks=1 txs=107 mgas=12.136 elapsed=1.378s mgasps=8.803 snapdiffs=5.99MiB triediffs=170.10MiB triedirty=48.30MiB