btcsuite
seems btcd is way slower than bitcoin core in syncing?
I remember I used bitcoin core to sync btc full node on EC2 (200+G) in one day. But I used btcd to sync full node -- two days later it just synced 108G ?
The EC2 config is the same.
@lzl124631x there are couple reasons that btcd is slower than bitcoind. The main reason I see is that btcd download from one peer only in bootstrapping the blocks, while bitcoind download from 8. So you need to find a closer and faster node to your ec2 instance. Let me know if you need more details, I can share what I did to speed up my syncing of 200+ G.. But it will never be as fast as bitcoind.
dcrd has an issue open related to multipeer downloads https://github.com/decred/dcrd/issues/1145 which appears to be partially done.
I haven't been following it closely enough to know if it is likely to be backportable to btcd but might be interesting to take a look at.
@lzl124631x : If your node is in US (like mine), your connection will be slow to the default nodes in btcd DNSSeed under chaincfg/params.go . If you just manually add couple nodes in US, the bootstrapping speed will be much faster. For example add this to your startup btcd command:--connection=157.131.198.183:8333 --connection==209.108.206.229:8333 . The IPs keep changing, you can use the leaderboard to find latest IPs close to your country.
I put more details in this post, if you want to look.
@totaloutput Good to know that! But is bitcoind smart enough to sync with nodes geographical closer? If yes, why doesn't btcd have that yet?
@lzl124631x I don't think current bitcoind code is smart enough to find geographically closer nodes to sync. (I could be wrong) The current code is to find 1 node to get the headers of all the blocks, and find 8 nodes to ask for the content (data) of the blocks, if any of the node is slow/unresponse for 2 seconds, it will drop that node and ask for a new one.
btcd code doesn't have that build in yet. It's a good feature, especially when the blockchain data is getting bigger and bigger. But majority of the btcd development resource moved onto dcrd (decred). My guess is that feature will be first implemented there then backport to btcd if someone wants to do it.
I am using the btcd codebase to update the parallelcoin network (it almost even still has malleability vulnerabilities) and looked at the p2p syncmanager stuff.
I made a small alteration that favours selecting the lowest ping out of the connected peers, the code is here: https://github.com/btcsuite/btcd/blob/master/netsync/manager.go#L261 where you need to make the change and here: https://github.com/btcsuite/btcd/blob/master/netsync/manager.go#L261 - I personally would think that more metadata stored about peers could get a more accurate picture of available block sources, such as a record of the last block/second rate when the peer was synced from. Still, lowest ping of up to 8 currently connected nodes probably still helps.
btcd as delivered in the git repo has a series of checkpoints. When checkpoints are set, it pre-fetches the block headers and compact filters (if available). This likely actually slows things down during initial sync. I'd suggest trying to disable the checkpoints, to eliminate this double-downloading. Checkpoints are mainly to stop miners filling mempools with new blocks forked from ancient blocks, which are extremely unlikely to ever exceed the cumulative work total of the current best block. With my work on a forked chain, I am finding that when I set checkpoints, everything after gets orphaned. I am looking into this.
Since checkpoints main purpose, in effect, is simply to reduce time-wasting old forks from clogging up the mempools, it's my opinion that the more effective strategy would be to simply disallow forks that would require an inordinate amount of hashpower to push ahead of the tip. Someone mentioned this a long time ago in a btctalk post, the idea of simply disallowing new blocks to attach before some reasonable number of blocks previously. I think it should be on the basis of cumulative weight. A reasonable figure could be derived from the known hashpower for an algorithm as currently exists on all networks.
And yes, the btcd netsync library only downloads from one source at a time. I think it would make sense to improve this when it is syncing pre-checkpoint to establish several links and download several segments of the chain from several sources at once, while grabbing the blocks-only from a few nodes at the same time. The consensus chain could then be established earlier in the sync process and of course nonconforming blocks, the node serving them up should get a huge banscore increment for this, i mean, weeks of no contact, trying to propagate a fake chain is pretty serious.
This is a huge bug. I am already waiting a week for BTCD to get all the blockchain. And this should really only take a day maximum. BitcoinD is for sure 20x faster. How can one roll out any service if the bitcoin daemon is so slow to get started?
@lzl124631x : If your node is in US (like mine), your connection will be slow to the default nodes in btcd DNSSeed under chaincfg/params.go . If you just manually add couple nodes in US, the bootstrapping speed will be much faster. For example add this to your startup btcd command:--connection=157.131.198.183:8333 --connection==209.108.206.229:8333 . The IPs keep changing, you can use the leaderboard to find latest IPs close to your country.
I put more details in this post, if you want to look.
I set connect=127.0.0.1, but it took three days to sync to 333365, and it's getting slower and slower.
I'm getting about 1 block every 2 seconds. I tried modifying same addpeer and connect config parameters a few times but didn't see any change.
It was much faster than this for the first 100,00 or 200,000 blocks. Slowed down in the mid-high 200.000's ...
@justinmoon Same to me. At first, 200 or 300 blocks every 10 seconds; after 178411, less than 100 blocks every 10 seconds; now, somtimes 1 block every 10 seconds. Kick-ass!
@justinmoon Same to me. At first, 200 or 300 blocks every 10 seconds; after 178411, less than 100 blocks every 10 seconds; now, somtimes 1 block every 10 seconds. Kick-ass!
What version are you running? I pulled roasbeef/btcd, installed latest, but ended up with 0.9.0-beta, which had a lot of stalled sync problems, was processing 1 block every 2 minutes at mid 300k.
Trying 0.12.0-beta now, I was doing 1k per 10s, but just passed 180k and I'm barely doing 200 per 10s now.
@OlaStenberg I'm running master(version 0.12.0-beta).
Ok, seem to be working a lot better for me with master, even if it slowed down to 20 blocks every 10s (around 350k). At what height did it slow down to 1 block every 10s for you?
@OlaStenberg I started syncing from the begining again. At first, 17k blocks per 10s, now(204518) about 40 per 10s. And I find it's "fetchUtxosMain" that's so slow. Maybe there are some design flaws.
Btcd sync time is around 10x slower than bitcoin core 0.18 for now.
I did short profiling of the code in sync state and found that the incredible amount of CPU time is wasting by the garbage collector.
This is in turn the result of the LevelDB calls. In the same time Btcd is using very low amount of the memory. In nowadays even a small devices like Raspberry PI has enough amount of the memory.
I thing the right way of optimization is to improve the algorithm of LevelDB handling and to optimize malloc usage
For me it was dealbraker and I switched to bitcoin core :( I synced whole blockchain in little more than day. With btcd after few days I was still in the 60% of the height and like 30% of the size. :( It makes me sad, I would like to see more diversity but with this, I have no other option :(
Two things stand out relating to this from my close work with the code on an old bitcoin fork (parallelcoin). One is the EC library. First thing I see is that the precompute could optionally be made a lot bigger, to improve performance, and on the other hand, also, using the fastest possible C library for koblitz curves optionally.
Secondly I don't think that all of the problem with the database is the leveldb back end, though there is probably issues there (again maybe cgo option?) and in between that and implementing the interface, common code exists also. This issue is common for Go. I watched a talk a few weeks ago about a video streaming service where they found they basically had to single thread the scheduling of the streams to reduce blocking, which of course points straight at the GC being incorrect for the task. It shouldn't be mysterious because the scheduler's main workload overall is IO bound and that's a bad fit for CPU bound stuff like transaction validation and database cache management.
Solving the problem probably will follow a pattern similar to the solution used by the streaming service, to explicitly schedule processing instead of trusting the scheduler, and minimising post init memory allocation with one big early allocation. All of which are a lot of work. The GC percent is default set to 10% and this is quite generous but still puts a ceiling on throughput, which conflicts with latency, in almost all cases.
In my profiling, garbage collection and runtime operations were taking up a lot of CPU time during sync, which was worrying - so I profiled some more (for allocations), and immutable treap operations were by far the biggest allocators, so that may be the issue.
Here's what I have for allocations when syncing 2012 blocks:
File: btcd
Type: alloc_space
Time: Aug 16, 2019 at 2:53pm (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10
Showing nodes accounting for 182.87GB, 85.76% of 213.25GB total
Dropped 379 nodes (cum <= 1.07GB)
Showing top 10 nodes out of 109
flat flat% sum% cum cum%
83.99GB 39.39% 39.39% 83.99GB 39.39% github.com/btcsuite/btcd/database/internal/treap.cloneTreapNode
64.99GB 30.48% 69.86% 65GB 30.48% github.com/btcsuite/btcd/txscript.parseScriptTemplate
6.75GB 3.16% 73.03% 34.08GB 15.98% github.com/btcsuite/btcd/database/internal/treap.(*Immutable).Delete
6.34GB 2.97% 76.00% 6.34GB 2.97% github.com/btcsuite/goleveldb/leveldb/table.(*Reader).newBlockIter
5.06GB 2.37% 78.37% 10.78GB 5.05% github.com/btcsuite/goleveldb/leveldb/util.Hash
4.92GB 2.31% 80.68% 4.92GB 2.31% bytes.NewBuffer
3.25GB 1.52% 82.20% 3.25GB 1.52% github.com/btcsuite/btcd/database/internal/treap.newTreapNode
2.87GB 1.35% 83.55% 2.87GB 1.35% github.com/btcsuite/btcd/wire.(*MsgTx).BtcDecode
2.52GB 1.18% 84.73% 16.38GB 7.68% github.com/btcsuite/btcd/blockchain.(*UtxoViewpoint).addTxOut
2.20GB 1.03% 85.76% 2.20GB 1.03% github.com/btcsuite/btcd/database/internal/treap.(*parentStack).Push
For cloneTreapNode:
(pprof) list cloneTreapNode
Total: 213.25GB
ROUTINE ======================== github.com/btcsuite/btcd/database/internal/treap.cloneTreapNode in /home/dan-server/btcd/database/internal/treap/immutable.go
83.99GB 83.99GB (flat, cum) 39.39% of Total
. . 14: return &treapNode{
. . 15: key: node.key,
. . 16: value: node.value,
. . 17: priority: node.priority,
. . 18: left: node.left,
83.99GB 83.99GB 19: right: node.right,
. . 20: }
. . 21:}
. . 22:
. . 23:// Immutable represents a treap data structure which is used to hold ordered
. . 24:// key/value pairs using a combination of binary search tree and heap semantics.
And parseScriptTemplate:
(pprof) list parseScriptTemplate
Total: 213.25GB
ROUTINE ======================== github.com/btcsuite/btcd/txscript.parseScriptTemplate in /home/dan-server/btcd/txscript/script.go
64.99GB 65GB (flat, cum) 30.48% of Total
. . 193:
. . 194:// parseScriptTemplate is the same as parseScript but allows the passing of the
. . 195:// template list for testing purposes. When there are parse errors, it returns
. . 196:// the list of parsed opcodes up to the point of failure along with the error.
. . 197:func parseScriptTemplate(script []byte, opcodes *[256]opcode) ([]parsedOpcode, error) {
64.99GB 64.99GB 198: retScript := make([]parsedOpcode, 0, len(script))
. . 199: for i := 0; i < len(script); {
. . 200: instr := script[i]
. . 201: op := &opcodes[instr]
. . 202: pop := parsedOpcode{opcode: op}
etc. etc. nothing else significant
So these combined account for about 70% of allocations, but account for less than 10% of in-use space at runtime. In both cases, keeping some state so we reduce the number of allocations (and deallocations) would be beneficial. I bet replacing the immutable treaps in the dbcache would really help sync speed. LevelDB calls are also fine, they are all fairly lightweight - IMO the issue is the cache. More profiling probably needs to be done, but my guess is the dbcache.
I'm trying to set up my own node with btcd instead of bitcoin core, since I want to support a healthy blockchain ecosystem. But after two weeks trying to sync the mainnet blockchain, it's just way too slow. I can't recommend anyone use this software when it takes so long just to get started. Sync times are increasingly slow as the block height rises - see the attached image.
(The gap on nov. 11 - 13 was from when I stopped syncing to mess with configs)
I will be focusing on solving this problem in my project (github.com/p9c/pod), but thanks to Rjected I also have looked at some of the treap code as well as the script engine and I am pretty sure both have got serious garbage accumulation problems. Optimising them and aiming for zero runtime allocation will probably go a long way towards a solution.
Incidentally, a part of btcd that I have had INTENSE work with is the CPU miner. It uses two mutexes, and stops and creates new goroutines constantly. I built a library that allows me to attach an RPC to the standard output, built a small, dedicated single thread (two threads but primary work), I use two channels, stop and start, and I thought I would need an atomic or compareandswap but it turns out just using two channels and a for loop with a 'work mode' second loop, both parts of the loop drain the channels not relevant (the runner ignores the run signal channel and the pauser ignores the pause signal channel), and the lock contention is obviously so bad that a little over 10% of potential performance is chewed up with synchronisation.
I know well enough from what I saw of the script engine and database drivers/indexers that the programmers who write it are obviously former C++/Java programmers because they pretty much mainly rely on mutexes for synchronisation and mutexes, which are the slowest sync primitives, and where channels are used, in places that I would expect to see async calls used in these older, less network-focused languages.
For bitcoin forks, especially small, neglected ones like parallelcoin, its sync rate is fine. 8 minutes on my Ryzen 5 1600/SSD/32gb machine, at a height of about 210,000. The chain barely has maybe 2 transactions per block, on average. But even still, at 99000 and again around 160000 it bogs down badly and appears to be mainly garbage collecting, so with the typical block payload of bitcoin, I imagine the complexity of the chain of provenance of tokens explodes exponentially, and that graph exactly shows this pattern.
I'm not sure where I will start with it, but I strongly suspect write amplification is also hiding in there, a performance problem well known to be existing with LevelDB, and even RocksDB and BoltDB, and resolved in Badger, so first step will be building a badger driver. I'd guess that especially as the number of transactions grows that write amplification is causing an issue every time the database updates the values it has to write the keys again as well, combined with the geometric rise in complexity of validations to confirm they correctly chain back to the coinbase.
Second thing I expect to look at is the treaps. There is some parts of btcd that attempt to eliminate runtime allocations, at least one buffer freelist, but there is a lot of creation of small byte slices that are discarded later. As tends to be the case with Go, the naive (I did mention the mutexes, they are a naive use of Go) implementation does not take into account GC or thread scheduling, and when the bottlenecks are really bad, usually it means you have to take over both memory and scheduling work from the Go runtime to get a better result. I already saw one clear example of this just in the use of isolated processes connected via IPC pipes and using two channels and one ticker instead of 2 or 3 mutexes and 3 different tickers made it produce more than 10% more hashes.
If anyone is interested who is following, keep an eye on the repo I mentioned above as over the next 6 months I will be focusing on optimizing everything. I am nearly finished implementing the beta spec for my fork, and I have aimed to make it accessible enough and not stomping over top of too much of what is already there that differs for the chain I am working on. I am still a bit lost as to how to enable full segwit support, and you will see I have merged btcwallet, btcd, btcutil and btcjson repositories into one, created a unified configuration system, and mostly done and robust handling of concurrent running of wallet and node together for a more conventional integral node/wallet mode of operation. I understand some of the reasons behind so vigorously separating them but in my opinion in the absence of a really good SPV implementation makes doing this a step backwards.
Based on watching CPU utilization and a little bit of profiling I can see so much empty spaces between with CPU doing literally nothing for about 60-70% of the time, during the sync process, so I am very leaning towards the idea that synchronization is the bigger issue, and second is garbage generation, and thirdly, write amplification due to the database log structure, with updating metadata related to block nodes in the database.
By the way, out of curiosity I tried disabling the addrindex and txindex while on initial sync. Not only did it not seem to take any less time, the amount of metadata generated in the database folder looked exactly the same as though it was on. I am not sure if it is ignoring my configuration change, I checked through all the places where that setting is read and it definitely should not have been running the indexers.
Looks like a memory ballast could be in order... https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/
I remember reading this one before. I am going to try it out with my fork at https://github.com/p9c/pod and I will report back if it gets that kind of result (it is on a different, much smaller change but even still appears to have at least one big GC cleanup every 50-100,000 blocks and that is at least part of the problem for sure, as one or two blocks end up taking like a minute to process.
Hey @l0k18, what ended up happening? If it worked I'd be happy to port here since this has become a much bigger issue
So the ballast (10gb) does appear to substantially increase the garbage collection rate:
Ballast:
No ballast:
It looks like the greatest decrease here comes from:
Ballast without debug.SetGCPercent(10):
No Ballast without debug.SetGCPercent(10):
This appears to result in a 3x increase in heap size:
10% GC limit w/ ballast:
10% GC limit w/ no ballast:
No Limit no ballast
No Limit
I'll need to do a full sync to confirm, but it looks like a 3x increase in memory can get us a 3-4x increase in block sync speed. Ballast does not appear to improve allocation speed
I have been working on a fork coin using this engine and I found it maxed out about 2000tx/s syncing from a node on my lan. I suspect this may be structural limit caused by signature verification performance. It has been a couple months since i was working on it so my memory might be hazy but i think it got best performance with gc limit at 100% which i think is 'no limit'.
I hadn't tried the ballast, and in a couple of days i can confirm.
Yeah, that's what I'm thinking. I think this is going to come down to a memory/cpu trade off. Maybe we can add a config option in here
since only vps nodes may need to squeeze into under 1/2gb and nearly no home users have less than 4gb it is an option only needed for the former case but it is not difficult to defer the gc/ballast settings after flags/config are parsed.
For comparison, what is the bitcoin core memory usage anyhow? 2k tps is decently fast so long as network isn't the bottleneck. It also will indicate how many bytes the p2p side needs to be delivering to keep the cpu humming
Yeah, so I think the goal here is to make network the bottleneck. Tough to do, but that's probably the stretch goal.
Bitcoin Core recomends 2gb of ram. I've personally never seen it use that little, our current node is about 5gb