neutrino icon indicating copy to clipboard operation
neutrino copied to clipboard

Published 20 hours ago • verified publisher icon lightninglabs

neutrino+query+rescan: improve rescan speed

Open ellemouton opened this issue 3 years ago • 5 comments

This PR should probably be split into 3 PR's. Doing it in 1 for now just to get some initial feedback and to better show the progression. The three sections are as follows:

  1. The first 2 commits alter the GetBlock and GetCFilter functions to use the work dispatcher for their queries instead of using the old queryPeers function. The function is now removed bringing us one step closer to removing all query logic from the main package.
  2. The 3rd commit isolates the bottleneck of the GetCFilter function which is persisting filters to the DB. In this commit, this operation is spun off into a goroutine thus allowing the GetCFilter function to return as soon as all the filters are written to the cache.
  3. Finally, the 4th commit ensures that rescan can make use of batch filter fetching by waiting until the header chain is either current or until it is ahead of the specified end height. Before this commit, if rescan is started before the chain is current, then the filters are fetched one by one instead which is what makes things super slow. For example, on local regtest: before this commit it would take 7 seconds to sync 3000 blocks but with this commit, it takes 900ms. (full testnet rescan from local testnet bitcoind took like 5 mins)

ellemouton avatar Nov 11 '21 10:11 ellemouton

Thanks for taking a look @positiveblue!!! Will address your comments soon 👍

ellemouton avatar Nov 19 '21 08:11 ellemouton

@ellemouton, remember to re-request review from reviewers for your latest update

lightninglabs-deploy avatar Dec 07 '21 15:12 lightninglabs-deploy

!lightninglabs-deploy mute

ellemouton avatar Dec 07 '21 15:12 ellemouton

Gave it a test on mainnet and there were no more single cfilters requests, just big chunks of blocks, which is great. Not always 1000, but never just 1.

chappjc avatar Dec 10 '21 22:12 chappjc

awesome! Thanks for testing @chappjc!

ellemouton avatar Dec 11 '21 05:12 ellemouton

hey guys, is anyone at lightninglabs looking into this? would really appreciate it! resyncing takes forever, and makes user experience really, really bad.

losh11 avatar Jan 14 '23 23:01 losh11

Closing - gonna open 3 follow-up PRs 🤓

replaced by https://github.com/lightninglabs/neutrino/pull/273, https://github.com/lightninglabs/neutrino/pull/274 and https://github.com/lightninglabs/neutrino/pull/275

ellemouton avatar May 05 '23 13:05 ellemouton