go-spacemesh icon indicating copy to clipboard operation
go-spacemesh copied to clipboard

fetch: add streaming mode

Open ivan4th opened this issue 1 year ago • 1 comments

Motivation

Excess buffering during sync causes unneeded GC pressure and sometimes leads to out-of-memory errors.

Description

Added streaming mode for the fetcher (feature-gated).

In streaming mode, no extra buffering is used for fetch server and client. Fetch client still uses some big slices such as list of ATX IDs in an epoch, but that's to be improved in a followup.

Streaming mode is enabled via fetch.streaming boolean config key, which defaults to false.

Test Plan

Verified sync on the testnet.

ivan4th avatar Feb 13 '24 10:02 ivan4th

Codecov Report

Attention: Patch coverage is 79.42446% with 143 lines in your changes are missing coverage. Please review.

Project coverage is 80.0%. Comparing base (b09651a) to head (da777b2). Report is 3 commits behind head on develop.

Files Patch % Lines
fetch/handler.go 64.0% 27 Missing and 19 partials :warning:
fetch/fetch.go 77.1% 19 Missing and 8 partials :warning:
p2p/server/server.go 73.1% 19 Missing and 7 partials :warning:
codec/codec.go 52.9% 7 Missing and 1 partial :warning:
datastore/store.go 83.3% 6 Missing and 2 partials :warning:
fetch/mesh_data.go 89.7% 4 Missing and 3 partials :warning:
sql/database.go 91.4% 2 Missing and 2 partials :warning:
sql/identities/identities.go 86.2% 2 Missing and 2 partials :warning:
sql/layers/layers.go 88.2% 2 Missing and 2 partials :warning:
activation/activation.go 50.0% 0 Missing and 2 partials :warning:
... and 4 more
Additional details and impacted files
@@            Coverage Diff            @@
##           develop   #5562     +/-   ##
=========================================
+ Coverage     79.8%   80.0%   +0.2%     
=========================================
  Files          279     279             
  Lines        28549   29009    +460     
=========================================
+ Hits         22789   23223    +434     
+ Misses        4186    4184      -2     
- Partials      1574    1602     +28     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Feb 13 '24 10:02 codecov[bot]

(almost forgot) Will further update this PR to move some of the codec stuff to go-scale, otherwise good for review

ivan4th avatar Feb 27 '24 10:02 ivan4th

bors try

ivan4th avatar Mar 11 '24 21:03 ivan4th

try

Build failed:

spacemesh-bors[bot] avatar Mar 11 '24 21:03 spacemesh-bors[bot]

bors try

ivan4th avatar Mar 11 '24 22:03 ivan4th

try

Build succeeded:

spacemesh-bors[bot] avatar Mar 11 '24 22:03 spacemesh-bors[bot]

bors merge

ivan4th avatar Mar 16 '24 18:03 ivan4th

bors cancel

ivan4th avatar Mar 16 '24 18:03 ivan4th

Canceled.

spacemesh-bors[bot] avatar Mar 16 '24 18:03 spacemesh-bors[bot]

bors merge

ivan4th avatar Mar 16 '24 20:03 ivan4th

Pull request successfully merged into develop.

Build succeeded:

spacemesh-bors[bot] avatar Mar 16 '24 21:03 spacemesh-bors[bot]