bitpay
Is there the mongodb database ready to download somewhere ?
I am trying to get started with your project, I've started to synchronize the database many days ago, like two weeks or so.
Since the beginning, the synchronization drop day after day. I am currently at 2 block per minutes which will lead at this rate if it does not go any slower to an additional 92days. I am currently on the 329990 block height, and I really want to be able to start related project Insight.
The server have this kind of good spec : Intel Xeon E3-1230v6 - 32GB DDR4 ECC 2133MHz So I was wondering if there is a problem on my server, something I could optimize to run this much faster (I would be glad if somebody can lead me to some related resources) or is there somewhere the Mongodb database files ready to download (Even if it is not up to date) with a block height close enough to the current 594363 height.
Thank you
Are you on platter drives? NVMe makes a huge difference. Is your bitcoin peer on the same system? It can take a few weeks to sync, but shouldn't be 92 days.
P.S. Periodically back up your mongodb. I can't tell you how frustrating it is if it goes bad, and you have to start all over.
Thank for taking the time to answer. Indeed i am on a platter drive. Sad to hear that would make the difference. On the other side the bitcoin peers is in the same drive. From what I understand you may have a mongodb ready, but on a strictly technical aspect, If somebody share with me with their mongodb files, do you confirm it would be only the matter of the config line in the mongod.conf line to get my bitcoin core "almost"ready to use ?
@needbill can you email me, jamespharvey20+bitcore at gmail dot com? Leave the + as is, just substitute the at and dot. I can offer something probably not appropriate to post here.
A completely synchronized bitcoin-core and bitcore approaches 1TB. Creating bitcore's mongodb is extremely intensive. I'm sure the bitcoin peer being on the same platter drive will slow it down even further. You should see a speedup splitting them, but if I had to guess, I don't think it would be a noticeable speedup. If I understand correctly, the bitcoin peer reading is fairly linear because bitcore is going block per block, and it's just the mongodb writing that is very random, so I wouldn't be surprised if it was at most a 5% speedup, but who knows without trying. Giving the process 32 cores, 64GB of memory, and a Samsung 970 NVMe EVO, the NVMe still seems to be the bottleneck.
If someone shared with you their entire bitcoin peer files and their mongodb files, and both were fully synced, you should be within minutes of having bitcore ready to use. There are things that could cause delays. If the mongodb files were made with a newer version of mongodb than you have, the files could be incompatible, forcing you to upgrade. If the bitcoin peer files were newer, you could have the same issue.
If you have an already synced bitcoin peer, I don't think you'd need the bitcoin peer files that was used to generate the bitcore mongodb database. I haven't seen that be mentioned, but I haven't tried switching an already synced bitcore to a different peer. Internally, a bitcoin peer's files are specific - you can't mix and match files from different peers. But, if I understand correctly, bitcore connects to the peer as just another peer, so it shouldn't be different than how all the peers get blocks from each other anyway, so I don't think it would matter.
EDIT-ADD: I'll also note that as you get deeper into the blocks, they get substantially slower. You probably saw it's slowed down over time, and unfortunately that trend continues and gets much worse. My rig gets around 90-100 blocks per minute around 350000 block height, and in the 590000 height range, about 19 blocks per minute.
So, you'd also have to consider that if the average is a new bitcoin block added to the blockchain every 10 minutes, that once synced, your system would have to be able to keep up with that. IF you saw an 80% slowdown like I do, and I don't know what yours would do, you might see 0.4 blocks per minute, which should keep up with new blocks. But, if you had it off for a while, it could take a considerable amount of time to get caught up again.
EDIT-ANOTHER ADD: You also need to consider that if you use anyone's copied bitcoin peer or bitcore mongodb database, there's the chance they could have done something nefarious with the data. I don't know how practical it would be to attack someone this way, but I've always seen warnings about downloading others' bitcoin peers, so wanted to forward that along.
Thank you for this very comprehensive answer. Kind of reassuring. Was wondering why no "authority" weren't providing such database, but the space it takes, the database versioning and the fact that people wont even trust the database could make sense. I will try to dig on the SSD side to get better results I guess.
I've got the whole btc and doge database by bitcore-node,if anyone want,I can afford it. Here is my email: [email protected]
