archives icon indicating copy to clipboard operation
archives copied to clipboard

Published 20 hours ago • verified publisher icon ipfs-inactive

Making sure archives are fetchable

Open victorb opened this issue 7 years ago • 11 comments

To make sure all archives are still online and can be fetched, I'm currently running through them with refs -r.

  • [ ] cdn.media.ccc.de | /ipfs/QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb (@lgierth )
  • [ ] arXiv | /ipfs/QmfXH9XtP7xmoTH8WAp4HNSduqWMwLTH8B8TvbTkdgzNAa
  • [x] IETF RFC Archive | /ipfs/QmNvTjdqEPjZVWCvRWsFJA1vK7TTw1g9JP6we1WBJTRADM
  • [x] Presidental Daily Briefs | /ipfs/Qme6epvZDj3vzHcFKdF1nZhbixjw8Bn4imGcKnbUyBJL89
  • [ ] PGP Public Key Database | /ipfs/Qmbs2DxMBraF3U8F7vLAarGmZaSFry3vVY5zytuN3BxwaY
  • [x] XKCD | /ipfs/Qmb8wsGZNXt5VXZh1pEmYynjB6Euqpq3HYyeAdw2vScTkQ
  • [ ] Online Encyclopedia of Integer Sequences | /ipfs/QmbXX6jkJSx1aH41nfBqPyZAgxe1m4CzMMVnhYz1xRyFDQ
  • [ ] scholarpedia.org | /ipfs/Qmaskk1Egq5zmZsGTd7dwNiiK1cwfmx7k1StG1WJQjwGDm
  • [x] No-Intro Collection | /ipfs/QmTC8ADX28gEWH68SsuduWoropCad9g9bfK2b2WWwYggPv
  • [ ] cdnjs | /ipfs/QmRrnfFUgx81KZR9ibEcxDXgevoj9e5DydB5v168yembnX
  • [ ] alpine-linux 3.4 packages | /ipfs/QmRsvEpJggeu4HhoafzRFobV4sbwVVTXMrdb2p8XWv7bCS
  • [x] Project Apollo Archives | /ipfs/QmSnuWmxptJZdLJpKRarxBMS2Ju2oANVrgbr2xWbie9b2D
  • [ ] textfiles.com | /ipfs/QmNoscE3kNc83dM5rZNUC5UDXChiTdDcgf16RVtFCRWYuU

victorb avatar Jul 30 '18 12:07 victorb

These seems dead, can only get the listing of first level links but gets no further:

  • Qmbs2DxMBraF3U8F7vLAarGmZaSFry3vVY5zytuN3BxwaY
  • QmbXX6jkJSx1aH41nfBqPyZAgxe1m4CzMMVnhYz1xRyFDQ
  • Qmaskk1Egq5zmZsGTd7dwNiiK1cwfmx7k1StG1WJQjwGDm

victorb avatar Jul 30 '18 12:07 victorb

cdn.media.ccc.de | /ipfs/QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb (@lgierth )

This one is also currently dead since it lives on a node whose badger datastore became unbootable. Working with the badger people to fix it.

ghost avatar Jul 30 '18 14:07 ghost

Is this just to get a report? Or do you intend on pinning this stuff on protocol-labs hardware?

eminence avatar Jul 30 '18 17:07 eminence

Mostly it's just to weed out things that went MIA, before DWeb Summit

ghost avatar Jul 30 '18 17:07 ghost

Although it might not apply here, note that refs -r may not fetch the leaves if they are raw.

kevina avatar Jul 30 '18 18:07 kevina

Oh what

ghost avatar Jul 30 '18 18:07 ghost

But raw leaves can have references in them? Think about Git objects and the like

ghost avatar Jul 30 '18 18:07 ghost

But raw leaves can have references in them?

No they can't. They are raw data with no structure and a leaf in the DAG tree. They by definition can not have links.

Think about Git objects and the like

I need examples.

kevina avatar Jul 30 '18 18:07 kevina

Should add these existing archives to index.html and the datasets section of awesome-ipfs:

  • Old Internet Files https://github.com/ipfs/archives/issues/176
  • ~~No Intro collection https://github.com/ipfs/archives/issues/163~~ already there
  • Web History Project https://github.com/ipfs/archives/issues/159
  • Pwned Passwords https://github.com/ipfs/archives/issues/157
  • MDSConnect https://github.com/ipfs/archives/issues/152
  • yarchive.net https://github.com/ipfs/archives/issues/76

ghost avatar Jul 30 '18 20:07 ghost

They're all added/removed in #178

ghost avatar Jul 31 '18 00:07 ghost

Think this is the right place, heads up the example given to try is also MIA. Can't fetch it all from cloudflare either.

Arxiv.org CC-By Papers: https://ipfs.io/ipfs/QmfXH9XtP7xmoTH8WAp4HNSduqWMwLTH8B8TvbTkdgzNAa/

h1z1 avatar Sep 19 '18 03:09 h1z1