bee icon indicating copy to clipboard operation
bee copied to clipboard

fix: add erasure reDecoder for evicted chunks

Open nugaon opened this issue 9 months ago • 0 comments

Checklist

  • [ ] I have read the coding guide.
  • [ ] My change requires a documentation update, and I have done it.
  • [ ] I have added tests to cover my changes.
  • [ ] I have filled out the description and linked the related issues.

Description

During long-lived joiner operations, reconstructed chunks can be evicted from the local cache due to memory pressure. When these chunks are needed again, the current implementation doesn't attempt to recover them a second time. Instead, it falls back to direct network fetching, which fails with ErrNotFound if the chunk isn't available in the network.

This creates a reliability issue where successfully recovered chunks become inaccessible once evicted from cache, potentially breaking long-running operations like downloads or uploads of large files.

This PR implements a erasure redecoder functionality with the following components:

  • ReDecoder: A new wrapper that attempts network fetch first, and only falls back to erasure recovery if the network fetch fails with ErrNotFound.
  • Lazy Decoder Instantiation: Recovery decoders are only created on-demand when network fetch fails, saving resources.
  • Memory-Efficient Caching: Maintains the existing memory optimization of nulling decoders after successful recovery while keeping track of successful recoveries.

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

nugaon avatar May 13 '25 15:05 nugaon