nano-node icon indicating copy to clipboard operation
nano-node copied to clipboard

Add reverse link store for fast receive block lookups by only their send block hashes

Open Exxenoz opened this issue 2 years ago • 6 comments

  • Added reverse_link_store to persist the [send block hash] -> [receive block hash] pairs.
  • Implemented LMDB v21 -> v22 migration to create the new reverse_links table on startup.
  • Implemented node.enable_reverse_links option (default disabled)
  • Hooked into block_cemented_callback to store the reverse links of new cemented blocks.
    • I'm still not sure if this is the best place to do that, but it seems to work.
  • Implemented initial reverse link pair loading into the node constructor.
    • I'm still not sure if this is the best place to start this relatively long loading process (it took around an hour on my system to process all cemented blocks from the live ledger), but it works.
  • Added some reverse link test cases
    • Test case for the initial reverse link loading in the nano::node constructor is missing, because I was not yet able to find a proper solution to modify/inject the nano::store before nano::node creation.

This PR will drastically improve the performance of #3702

Exxenoz avatar Mar 28 '22 17:03 Exxenoz

@dsiganos @thsfs Commit squash & rebase done (PR includes the iterable_store implementation from #3766).

Exxenoz avatar Mar 29 '22 21:03 Exxenoz

@dsiganos , @clemahieu , how the reverse links table could get affected by the confirmation height?

thsfs avatar Mar 30 '22 17:03 thsfs

I think the reverse_link table should be outside of the core database and in a separate auxiliary database. That will:

  • avoid the core database versioning problem
  • allow flexibility
  • separation of core data and auxiliary data

dsiganos avatar Mar 31 '22 11:03 dsiganos

I think the reverse_link table should be outside of the core database and in a separate auxiliary database. That will:

  • avoid the core database versioning problem
  • allow flexibility
  • separation of core data and auxiliary data

Oof, this would make the persistence layer even more complicated than it already is. In addition these changes would have to be made for both lmdb and rocksdb. I'm not really convinced it's worth the effort, especially if there are plans to move to an SQL backend.

Exxenoz avatar Apr 01 '22 21:04 Exxenoz

I will update the PR within the next two days. Have to finish another task first, but should be done soon.

Exxenoz avatar Apr 22 '22 17:04 Exxenoz

I will update the PR within the next two days. Have to finish another task first, but should be done soon.

Done. I also readded the LMDB reverse_links table creation and increased the store version.

Exxenoz avatar Apr 24 '22 20:04 Exxenoz