devalias.net icon indicating copy to clipboard operation
devalias.net copied to clipboard
verified publisher icon 0xdevalias

[DeepDive] IPFS, Distributed Hash Tables (DHT), Decentralised Storage, Torrents, etc

Open 0xdevalias opened this issue 6 years ago • 0 comments

Crypto, blockchain, etc related

  • https://ipfs.io/
    • https://en.wikipedia.org/wiki/InterPlanetary_File_System
  • https://storj.io/
  • https://sia.tech/
  • https://filecoin.io/
  • https://medium.com/bitfwd/what-is-decentralised-storage-ipfs-filecoin-sia-storj-swarm-5509e476995f

Dat

Dat is a protocol for sharing data between computers. Dat’s strengths are that data is hosted and distributed by many computers on the network, that it can work offline or with poor connectivity, that the original uploader can add or modify data while keeping a full history and that it can handle large amounts of data.

  • https://dat.foundation/ -> Community-driven project for distributed data synchronization
    • https://docs.datproject.org/
    • https://www.datprotocol.com/
    • https://datprotocol.github.io/how-dat-works/
  • https://github.com/datproject
    • https://github.com/datproject/dat

      • Use Dat command line to share files with version control, back up data to servers, browse remote files on demand, and automate long-term data preservation.

    • https://github.com/datproject/dat-store
    • https://github.com/datproject/sdk
    • https://github.com/datproject/dat-link-resolve
    • https://github.com/datproject/dat-node
    • https://github.com/datproject/dat-js
    • https://github.com/datproject/dat-http
    • https://github.com/datproject/datBase

      • Open data powered by Dat. Future-friendly apps for your research data pipeline.

      • https://datbase.org/
    • https://github.com/datproject/dat-storage
    • https://github.com/datproject/dat-secret-storage

mafintosh / Hyper*

  • https://github.com/hyperdivision

    • Scaling, distributed systems and crypto. dat://hyperdivision.dk

  • https://github.com/mafintosh
    • https://github.com/mafintosh/peerflix
    • https://github.com/mafintosh/peercast
    • https://github.com/mafintosh/playback
    • https://github.com/mafintosh/torrent-stream

      • torrent-stream is a node module that allows you to access files inside a torrent as node streams.

    • https://github.com/mafintosh/torrent-mount

      • Mount a torrent (or magnet link) as a filesystem in real time using torrent-stream and fuse.

    • https://github.com/mafintosh/torrent-blob-store

      • BitTorrent backed streaming blob store, both read and write supported

    • https://github.com/mafintosh/torrent-docker

      • realtime boot of remote docker images using bittorrent

    • https://github.com/mafintosh/hypercore

      • Hypercore is a secure, distributed append-only log

    • https://github.com/mafintosh/hypercore-protocol

      • Stream that implements the hypercore protocol

    • https://github.com/mafintosh/hyperdht
    • https://github.com/datproject/hyperdiscovery
      • https://github.com/mafintosh/discovery-swarm
      • https://github.com/RangerMauve/discovery-swarm-web
      • https://github.com/maxogden/discovery-channel
    • https://github.com/mafintosh/hyperdrive

      • Hyperdrive is a secure, real time distributed file system

    • https://github.com/datproject/hyperdrive-http

      • Serve a hyperdrive archive over HTTP.

DHT

  • https://www.reddit.com/r/torrents/comments/7wb4y0/possible_to_search_for_torrent_by_file_hash/

    • This is not exactly easy, but one thing I've considered doing is generating maybe 10-20 "likely" hashes, then checking the DHT. Assuming the torrent was public, didn't have any non-standard info fields, and the files have not been re-named, the only piece of information you're really missing is the "piece length". Since that's almost always a power of 2, and there are common "default" values, you could try generating some hashes and seeing if they work. You'd probably find at least something.

    • What I had in mind was to create a reverse index by taking magnet link dumps of various websites and fetching all the filehashes from the torrents. But I'm not completely sure this is possible. But once this is done, others could also make use of the reverse index.

  • https://www.usenix.org/conference/woot10/crawling-bittorrent-dhts-fun-and-profit

    • Crawling BitTorrent DHTs for Fun and Profit

  • https://github.com/maxogden/dhtkv

    • CLI for storing key/value data in the bittorrent mainline DHT

    • You can use get and put to use BEP 44 functionality to store arbitary 1KB immutable or mutable metdata in the DHT itself

    • http://www.bittorrent.org/beps/bep_0044.html

      • This extension enables storing and retrieving of arbitrary data in the BitTorrent DHT [1]. It supports both storing immutable items, where the key is the SHA-1 hash of the data itself, and mutable items, where the key is the public key of the key pair used to sign the data.

Bittorrent Protocol

BitTorrent Enhancement Proposals (BEPs)

  • http://www.bittorrent.org/beps/bep_0000.html

    • This document indexes all BitTorrent Enhancement Proposals (BEPs)

  • http://www.bittorrent.org/beps/bep_0005.html

    • DHT Protocol

  • http://www.bittorrent.org/beps/bep_0011.html

    • Peer Exchange (PEX)

  • http://www.bittorrent.org/beps/bep_0030.html

    • Merkle hash torrent extension

  • http://www.bittorrent.org/beps/bep_0033.html

    • specifies an extension to the BitTorrent DHT (BEP 5) to support scrapes via distributed counting based on bloom filters

  • http://www.bittorrent.org/beps/bep_0035.html

    • Torrent Signing

  • http://www.bittorrent.org/beps/bep_0038.html -> Finding Local Data Via Torrent File Hints
  • http://www.bittorrent.org/beps/bep_0039.html

    • Updating Torrents Via Feed URL

  • http://www.bittorrent.org/beps/bep_0042.html

    • DHT Security extension

  • http://www.bittorrent.org/beps/bep_0043.html

    • Read-only DHT Nodes

  • http://www.bittorrent.org/beps/bep_0044.html

    • Storing arbitrary data in the DHT

  • http://www.bittorrent.org/beps/bep_0046.html

    • Updating Torrents Via DHT Mutable Items

  • http://www.bittorrent.org/beps/bep_0049.html

    • Distributed Torrent Feeds

  • http://www.bittorrent.org/beps/bep_0050.html

    • Publish/Subscribe Protocol

    • This BEP defines a modification of the DHT protocol to provide a topic based publish/subscribe service rather than a key/value store.

  • http://www.bittorrent.org/beps/bep_0051.html

    • DHT Infohash Indexing

    • With this extension a single node should be able to survey the entire DHT within a few hours without having to resort to non-compliant behavior.

Unsorted

  • TODO: include other resources from old notes/etc here: decentralised netflix, etc
  • https://www.resilio.com/individuals/
  • https://webtorrent.io/
    • https://webtorrent.io/docs
    • https://webtorrent.io/desktop/
    • https://github.com/webtorrent
      • https://github.com/webtorrent/webtorrent
      • https://github.com/webtorrent/instant.io
      • https://github.com/webtorrent/bittorrent-dht
      • https://github.com/webtorrent/bittorrent-tracker
      • https://github.com/webtorrent/create-torrent
      • https://github.com/webtorrent/parse-torrent
  • https://github.com/ungoldman/magnet-link
    • http://magnet-uri.sourceforge.net/

0xdevalias avatar Nov 30 '19 03:11 0xdevalias