kubo icon indicating copy to clipboard operation
kubo copied to clipboard

Garbage Collection minimum data lifetime

Open kallisti5 opened this issue 2 years ago • 1 comments

Checklist

  • [X] My issue is specific & actionable.
  • [X] I am not suggesting a protocol enhancement.
  • [X] I have searched on the issue tracker for my issue.

Description

In #8103 we saw users attempting to pin "large" data sets (in this case, 90GiB) reach a hangup during the pinning process.

If the size of the dataset you're attempting to pin is larger then GCThreshold, the chunks will be immediately garbage collected when ctl+c'ing or terminating the pin. This defeats the purpose of IPFS as users should be able to "resume" downloading chunks from where they left off if pinning is interrupted.

A simple solution to this may be to preserve pinned chunks for a minimum amount of time? 24 hours since being received, 48 hours since being received, etc. This would give users the opportunity to "resume" a failed pin.

#8413 is another alternative strategy to fix this (warning users on ipfs pin) #3121 seems to be another alternative strategy for this (best effort pins)

kallisti5 avatar Sep 07 '21 13:09 kallisti5

We want to overhaul the GC mechanism entirely, so linking to #7752

guseggert avatar Aug 05 '22 15:08 guseggert