Bittorrent to distribute models amongst nodes

[austinbv]

I am opening an issue for discussion as well as a place holder for work where we use bittorrent to distribute large models amongst nodes.

Implementation Path

• Network should check to see if the model is downloaded on any of the existing nodes.
• If the model doesn't exist on any node one node should register itself as the Seed node
• Seed node should download the model from hugging face and distribute the model via bittorrent to other nodes
• As nodes get parts of the seed they should distribute their parts to other nodes
• If the model exists on one or more nodes those nodes should advertise as seeds
• New nodes to a network will leech from existing nodes rather than the internet

✅ Pros

• Allows for networks to minimize model downloads saving hundreds of gigs if not TB of data use
• Speeds up distribution of models in a network - after the seed node has fully downloaded the node all nodes will finish leeching the model around the same time
• Models are guaranteed to be the same b/c of bittorrent native hash checking