bitcore-node
bitcore-node copied to clipboard
bitpay
Extremely slow sync from local node
I'm running Bitcore-Node version 5.0.0 beta 8. I'm syncing from a local, fully synced Bitcoin Core 0.15 node. I've gotten to block 378864 and now it's taking on average 14 minutes to process each batch of 144 blocks, as high as 25 minutes. Is this normal? At this rate it will take me more than a week to sync the remaining blocks, and that's only if it doesn't slow down further.
Update: up to block 405216 and averaging 24 minutes per 144 blocks, maximum of 46 minutes to process blocks 404785 through 404928.
Estimated time remaining is 220 hours, but I think it's going to get slower...
I'm having the same issue as well, I tried syncing using bcoin first but it was also extremely slow.
Anyone able to get a node synced? Mine was almost done but it suddenly ran out of memory and now eats 8 GB of RAM and goes OOM during startup.
I'm also getting slow sync times. I was just reading some documentation on bcoin, and I found this quote here:
If the build fails compilation for bcoin-native or secp256k1-node validation will be slow (a block verification which should take 1 second on consumer grade hardware may take up to 15 seconds). Bcoin will throw a warning on boot if it detects a build failure. If you run into this issue, please post an issue on the repo.
I then checked my bitcore-node that I cloned from this repo:
$ ./bin/bitcore-node start | grep warning
[warning] (chain) P2SH has been activated.
[warning] (chain) BIP34 has been activated.
[warning] (chain) BIP66 has been activated.
[warning] (chain) BIP65 has been activated.
[warning] (chain) CSV has been activated.
[warning] (chain) Segwit has been activated.
[warning] (miner) No reward address is set for miner!
[warning] (node) Warning: secp256k1-node was not built.
[warning] (node) Verification will be slow.
I can't figure out how to get secp256k1-node to build.
@chrisrico, is this what's going on for you?
Sure enough, if I do an npm install in any bitcore-node v5 version, release or git clone, I get:
prebuild-install || node-gyp rebuild || echo "Secp256k1 bindings compilation fail. Pure JS implementation will be used."
...
prebuild-install http request GET https://github.com/cryptocoinjs/secp256k1-node/releases/download/v3.2.5/secp256k1-v3.2.5-node-v57-linux-x64.tar.gz
prebuild-install http 404 https://github.com/cryptocoinjs/secp256k1-node/releases/download/v3.2.5/secp256k1-v3.2.5-node-v57-linux-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=8.7.0 runtime=node arch=x64 platform=linux)
and indeed, node-v57 is not available for secp256k1-node v3.2.5 in the pre-built releases. However, v57 is available for secp256k1-node v3.3.0.
@kleetus, any recommendations?
If you don't have a preferred block source (trusted peer), Bcoin will be started automatically and synchronized with the mainnet chain.
So bcoin is not used if you are connected to a trusted peer?
I am using v5 and am connected to a local node but synch is extremly slow. Most blocks take 1 second but some take a few minutes. This will only get worse since I am around block 233xxx (~100kb blocks).
My current progress is 455746/532421 (85.5988%) for bitcoin cash (7 days of syncing). Bitcoin db - 159 GB, bitcore db - 303 GB. Looks like 5.x version is not usable for big blockchains like bitcoin or bitcoin cash.
I'm facing the same issue trying to sync from a local fully synced bcoin node. I'm on the v8 branch of bitcore, taking an average of 10 seconds per block at blockheight 325388 (it got progressively worse, I'm assuming due to the amount of transactions). Any ideas on how to fix this?
I am using v5.0 and I have been attempting to sync a regular Bitcoin bitcore-node server for about a month now. I was able to download the full chain using Bcoin in around a week, and at first the blocks were being processed quickly by bitcore node, but the block processing by "BlockService" after I hit ~block 400,000 goes at about 1 block per 2-10 minutes... So at this rate I will never finish processing the blocks.
I am currently at 495435/540376 (91.6834%), but each block takes 5-10 minutes now... I am assuming the slow down is because all the blocks it is now processing are all full to the brim, but it really shouldn't be going this slow.
bitcorenode.db is currently 431GB, and the Bcoin chain (chain.ldb) is currently 178GB.
Any idea on steps we could take in order to get our sync to finish/speed up? Making the BlockService use multiple threads maybe? Should we be using a different version (i.e. v4)?
got same problem, sync is going already 4 days (45%). Before i had bitcoind, its sync time was about 4-6 hours (400gb)
Same problem here... I have a fully synced bitcoind and now it's 2 days of syncing bitcore and I'm only at 292438 on a dual core 8GB machine.
same problem for me. using 32 cores and 32 GB ubuntu server. database is huge and extremely slow if block contains 1K+ transactions.
Hello, I have the same issue both on testnet3 and livenet. 4 days of syncing. Tried v8.1.1, v8.1.0, v8.0.0 Can anybody point me to the version that can think fast?
I found that bitcore opens a lot of connections to mongo. 67 connections after a 10 minutes of sync. Is it connected to this issue?
bitcoind cant affect this. data retrieval from bitcoind is instant, but saving to mongo is very slow. the speed is acceptable if block contains 500-700 transactions. for 1500+ its terrible. even on 32 cores cpu.
Wait until you try to sync BSV with it. If anyone is interested, there is an BSV Insight API available at https://www.mattercloud.net
Syncing with local bitcoin-core. Disabled journaling made me 20% faster. 12 blocks/min at 380000.
Oct 21 18:49:47 btcnode node[28890]: {"message":"2019-10-21 18:49:47.139 GMT+2 | Syncing... | Chain: BTC | Network: mainnet | 2.92 blocks/min | Height: 414715","level":"info"}
anybody managed to get it over 3blocks/min >400k height?
Added an nvme for db folder now im at 250/300 block minute at block 300000 im was at 14 block minute in hdd
Ubuntu 16.04 Node 8.16 Mongodb 4.2 (maybe downgrade mongod will solve this slow rate) Ram 64gb Cpu i7 7800x
After 8 days im at block 600k with 20/25 block by minute so i think more 1 day to full sync
Ubuntu 16.04 Node 8.16 Mongodb 4.2 (maybe downgrade mongod will solve this slow rate) Ram 64gb Cpu i7 7800x 1tb NVME
10 days full sync 600Go mongodb data base
Ubuntu 16.04 Node 8.16 Mongodb 4.2 (maybe downgrade mongod will solve this slow rate) Ram 64gb Cpu i7 7800x 1tb NVME
It took nearly six hours to troubleshoot the problem. The core of the problem was that disk IOPS limited the write efficiency of MongoDB, resulting in a very slow service. Optimization suggestions:
- Use storage media that support higher IOPS, for example, Provisioned IOPS SSD
- Rewrite the bitcore-Node project structure to add message queues to reduce disk IO
