ipwb
ipwb copied to clipboard
Enhancement: Add everything from a WARC under one pin instead of many
From @ProximaNova in #810, they suggested (summarized):
Could you provide an option for "ipwb index" that adds everything added from .warc under one pin instead of many pins. Use: for those who want to have as few CIDs in their pinset as possible. It wouldn't matter to those who don't care about having hundreds/thousands of pins. For now, I did this:
- edit .cdxj with vim to just get CIDs, on per line, save to a different file (don't overwrite the CDXJ)
- run
cat f1.cdxj.cid | xargs -d "\n" sh -c 'for args do ipfs files cp /ipfs/$args /a/f1cid/$args && ipfs pin rm $args; done' _ - run
ipfs files ls --long /athen pin the CID you see for that