ipwb icon indicating copy to clipboard operation
ipwb copied to clipboard

Enhancement: Add everything from a WARC under one pin instead of many

Open machawk1 opened this issue 1 year ago • 0 comments

From @ProximaNova in #810, they suggested (summarized):

Could you provide an option for "ipwb index" that adds everything added from .warc under one pin instead of many pins. Use: for those who want to have as few CIDs in their pinset as possible. It wouldn't matter to those who don't care about having hundreds/thousands of pins. For now, I did this:

  1. edit .cdxj with vim to just get CIDs, on per line, save to a different file (don't overwrite the CDXJ)
  2. run cat f1.cdxj.cid | xargs -d "\n" sh -c 'for args do ipfs files cp /ipfs/$args /a/f1cid/$args && ipfs pin rm $args; done' _
  3. run ipfs files ls --long /a then pin the CID you see for that

machawk1 avatar May 07 '24 16:05 machawk1