kubo
kubo copied to clipboard
Garbage Collection minimum data lifetime
Checklist
- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the issue tracker for my issue.
Description
In #8103 we saw users attempting to pin "large" data sets (in this case, 90GiB) reach a hangup during the pinning process.
If the size of the dataset you're attempting to pin is larger then GCThreshold, the chunks will be immediately garbage collected when ctl+c'ing or terminating the pin. This defeats the purpose of IPFS as users should be able to "resume" downloading chunks from where they left off if pinning is interrupted.
A simple solution to this may be to preserve pinned chunks for a minimum amount of time? 24 hours since being received, 48 hours since being received, etc. This would give users the opportunity to "resume" a failed pin.
#8413 is another alternative strategy to fix this (warning users on ipfs pin) #3121 seems to be another alternative strategy for this (best effort pins)
We want to overhaul the GC mechanism entirely, so linking to #7752