rust-lightning icon indicating copy to clipboard operation
rust-lightning copied to clipboard

Parallelize `ChannelMonitor` loading from async `KVStore`s

Open TheBlueMatt opened this issue 3 months ago • 19 comments

Reading `ChannelMonitor`s on startup is one of the slowest parts of
LDK initialization. Now that we have an async `KVStore`, there's no
need for that, we can simply paralellize their loading, which we do
here.

Sadly, because Rust futures are pretty unergonomic, we have to add
some `unsafe {}` here, but arguing its fine is relatively
straightforward.

Based on #4146 just cause I don't want to resolve conflicts.

TheBlueMatt avatar Oct 07 '25 23:10 TheBlueMatt

🎉 This PR is now ready for review! Please choose at least one reviewer by assigning them on the right bar. If no reviewers are assigned within 10 minutes, I'll automatically assign one. Once the first reviewer has submitted a review, a second will be assigned if required.

ldk-reviews-bot avatar Oct 07 '25 23:10 ldk-reviews-bot

Codecov Report

:x: Patch coverage is 47.58065% with 65 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 89.35%. Comparing base (765e43d) to head (f2b14fb). :warning: Report is 24 commits behind head on main.

Files with missing lines Patch % Lines
lightning/src/util/persist.rs 35.82% 38 Missing and 5 partials :warning:
lightning/src/util/native_async.rs 56.25% 14 Missing :warning:
lightning-block-sync/src/gossip.rs 0.00% 5 Missing :warning:
lightning/src/util/async_poll.rs 85.00% 3 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4147      +/-   ##
==========================================
- Coverage   89.36%   89.35%   -0.01%     
==========================================
  Files         180      180              
  Lines      139366   139712     +346     
  Branches   139366   139712     +346     
==========================================
+ Hits       124547   124843     +296     
- Misses      12227    12276      +49     
- Partials     2592     2593       +1     
Flag Coverage Δ
fuzzing 35.10% <0.00%> (-0.08%) :arrow_down:
tests 88.69% <47.58%> (-0.02%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Oct 07 '25 23:10 codecov[bot]

This doesn't actually work. monitor loading is (at least for a semi-local disk) CPU-bound, so we really need to spawn each monitor load task rather than having one task.

TheBlueMatt avatar Oct 08 '25 11:10 TheBlueMatt

This requires GATs which requires an MSRV bump (to 1.64), but we're planning on doing that soon anyway.

TheBlueMatt avatar Oct 09 '25 19:10 TheBlueMatt

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 13 '25 00:10 ldk-reviews-bot

🔔 2nd Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 15 '25 00:10 ldk-reviews-bot

🔔 3rd Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 18 '25 00:10 ldk-reviews-bot

🔔 4th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 20 '25 00:10 ldk-reviews-bot

🔔 5th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 22 '25 00:10 ldk-reviews-bot

🔔 6th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 25 '25 00:10 ldk-reviews-bot

🔔 7th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 27 '25 00:10 ldk-reviews-bot

🔔 8th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Oct 29 '25 00:10 ldk-reviews-bot

🔔 9th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Nov 01 '25 00:11 ldk-reviews-bot

🔔 10th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Nov 03 '25 00:11 ldk-reviews-bot

🔔 11th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Nov 05 '25 00:11 ldk-reviews-bot

🔔 12th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Nov 08 '25 00:11 ldk-reviews-bot

🔔 13th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Nov 10 '25 00:11 ldk-reviews-bot

Rebased on #4175, which avoids some further conflicts.

TheBlueMatt avatar Nov 10 '25 16:11 TheBlueMatt

Marking draft until the parent is marged.

TheBlueMatt avatar Nov 11 '25 21:11 TheBlueMatt

Okay this is ready now.

TheBlueMatt avatar Dec 08 '25 15:12 TheBlueMatt

🔔 1st Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Dec 10 '25 15:12 ldk-reviews-bot

🔔 2nd Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Dec 13 '25 00:12 ldk-reviews-bot

🔔 3rd Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Dec 15 '25 00:12 ldk-reviews-bot

🔔 4th Reminder

Hey @jkczyz! This PR has been waiting for your review. Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot avatar Dec 17 '25 00:12 ldk-reviews-bot

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

ldk-reviews-bot avatar Dec 19 '25 20:12 ldk-reviews-bot