nearcore icon indicating copy to clipboard operation
nearcore copied to clipboard

feat: prefetch more trie keys

Open Longarithm opened this issue 3 months ago • 3 comments

Longarithm avatar Mar 28 '24 20:03 Longarithm

Could the keys accessed via ft_balance_of be prefetched as well? Such access was quite frequent in Cosmose unit tests, though I don't know how relevant they are for mainnet traffic.

mooori avatar Mar 29 '24 10:03 mooori

Ah. I suspect ft_balance_of is a read-only query, and these ones are served from FlatStorage, which is only 1 fast disk read. Only write queries matter.

Note: with stateless validation everything will be served from memtrie in memory, so hopefully this hack is a short-term. We consider enabling memtries even sooner.

Longarithm avatar Mar 29 '24 10:03 Longarithm

Codecov Report

Attention: Patch coverage is 42.04545% with 102 lines in your changes are missing coverage. Please review.

Project coverage is 71.44%. Comparing base (57ff810) to head (964bb71).

Files Patch % Lines
runtime/runtime/src/prefetch.rs 31.75% 93 Missing and 8 partials :warning:
core/store/src/trie/config.rs 90.00% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10899      +/-   ##
==========================================
- Coverage   71.49%   71.44%   -0.06%     
==========================================
  Files         759      759              
  Lines      152522   152594      +72     
  Branches   152522   152594      +72     
==========================================
- Hits       109042   109015      -27     
- Misses      38964    39051      +87     
- Partials     4516     4528      +12     
Flag Coverage Δ
backward-compatibility 0.24% <0.00%> (-0.01%) :arrow_down:
db-migration 0.24% <0.00%> (-0.01%) :arrow_down:
genesis-check 1.43% <5.74%> (+<0.01%) :arrow_up:
integration-tests 37.12% <21.02%> (-0.06%) :arrow_down:
linux 69.92% <41.47%> (-0.02%) :arrow_down:
linux-nightly 70.89% <42.04%> (-0.07%) :arrow_down:
macos 54.46% <32.95%> (-0.04%) :arrow_down:
pytests 1.65% <5.74%> (+<0.01%) :arrow_up:
sanity-checks 1.44% <5.74%> (+<0.01%) :arrow_up:
unittests 67.07% <41.47%> (-0.05%) :arrow_down:
upgradability 0.29% <0.00%> (-0.01%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Mar 29 '24 12:03 codecov[bot]

Also do we have any good metrics and logs for prefetcher? If not can you add those and prepare a dashboard?

wacban avatar Apr 02 '24 09:04 wacban

@Longarithm could we merge this PR :)

bowenwang1996 avatar Apr 03 '24 23:04 bowenwang1996

FWIW we can look and keep an eye on the prefetching graphs to get some intuition as to whether the prefetcher is working as we intend it to. But also, we should merge this sooner rather than later? We've a release pending soon and if my memory serves me right we wanted this PR in it?

cc @VanBarbascu

nagisa avatar Apr 05 '24 13:04 nagisa

Addressed most comments, prioritising getting this into next release. Had some trouble with merging with prefetching logic on master but came through it.

Longarithm avatar Apr 05 '24 23:04 Longarithm