Integrate pinning / seeding daemon with CLI

[RangerMauve]

I'm using this to track progress on the work I'll be doing integrating a pinning / seeding service with the Dat CLI.

• [x] Figure out what to call it so it makes sense to everyone (twitter poll)
  • Going to go with pinning for now since that's the name of the DEP
• [ ] Identify what to use for the pinning
  • [x] Maybe dat-librarian
  • [x] hyperdrive
  • [ ] Use the header DEP
  • [ ] hypercore
  • [ ] hyperdb
• [x] Should integrate with DEP 0003 for an HTTP API (local only, should it have auth?)
• [x] Build a cli extension: dat-pin
  • [x] Persist credentials / service URL to FS
  • [x] Commands to talk to service
    • [x] set-service / unset-service / get-service
    • [x] login / logout
    • [x] add / remove / list
  • [x] Local pinning service
    • [x] start-service, install-service, uninstall-service
    • [x] Use os-service module
      • [ ] Linux
      • [ ] Mac
      • [ ] Windows
    • [x] Integrate with discovery-swarm-web so that the daemon also acts as a proxy for web apps.
• [x] Integrate with dat CLI pr
• [ ] Make sure it works with executible dat cli
• [x] Update docs with examples on how to use it
• [ ] Figure out a DEP for different data structures to identify public keys of additional feeds that need to be replicated
• [x] Add support for this DEP to the pinning service
• [ ] Add docs for spinning up Homebase using Docker or Digital Ocean / AWS

[joehand]

If you want to start with any of these as an extension we can add them, they just need to be published as dat-something and then whitelisted. Then can be run as top level commands dat something.

[RangerMauve]

I was thinking I'd need to modify the existing registry code so that dat publish would automatically target a daemon. Though launching the daemon could easily be a separate module.

[RangerMauve]

Here's the progress on the extension: dat-pin.

It's "working" locally, but I'll need to add unit tests and stuff next week before I release it.

[RangerMauve]

dat-pin is tested and released. Working on adding it to the CLI

[RangerMauve]

Bikeshedding discussion on IRC:

10:11 <@pfrazee> Paul haha
10:11 going to invite a couple other people. Bikesheds are brutal as open discussions
10:12 <rangermauve> RangerMauve I'll have some tissues on hand in case I break out into tears
10:12 @pfrazee set the topic: Bikesheds suck, let's do this real quick as a closed discussion
10:12 <@pfrazee> Paul ha
10:15 rangermauve: good work on the cli btw
10:15 <rangermauve> RangerMauve Thank you! It was handy that you already made the pinning client and garbados made dat-librarian. I just needed to add a bit of glue and persistence
10:22 → jhand joined (sid114474@gateway/web/irccloud.com/x-yujpcnodgzqjzuuj)
10:22 <@pfrazee> Paul https://gist.github.com/pfrazee/199bb8fe92f98ade6c259aa965232cdb
closepfrazee — 2 Apr 2019
10:23 @pfrazee set the topic: Bikesheds suck, let's do this real quick as a closed discussion. Current list: https://gist.github.com/pfrazee/199bb8fe92f98ade6c259aa965232cdb
10:23 <@pfrazee> Paul so my current ideas are "mirroring service" and "persistence service"
10:24 <rangermauve> RangerMauve I still really like "publishing to a resharing service" TBH
10:24 Persistence and mirroring are pretty good. They sound pretty techy though
10:24 What's your main concern with pinning?
10:25 <@pfrazee> Paul I don't think it's very descriptive
10:25 <rangermauve> RangerMauve I agree. It doesn't relate to what it's actually doing and why it's needed
10:26 <@pfrazee> Paul I'd like to be able to say to somebody "You need a X" and they be like "oh yeah makes sense"
10:26 You need a Permahost
10:26 <rangermauve> RangerMauve What kind of somebody are you targeting? I think that unless they already understand P2P, it'll be hard to find a word for it.
10:27 <@pfrazee> Paul Technical to semi-technical, with the potential for it to become vernacular
10:27 people understand what having a "cloud" means now
10:28 <rangermauve> RangerMauve And at the same time, if they already know about P2P, then they might already be comfortable with words from other projects like IPFS or BitTorrent
10:28 <@pfrazee> Paul like, if you said "oh no somebody hacked my cloud" people would understand that you had some service that hosted your data and it got broken into
10:28 rangermauve: right, that's why I was comfortable with pinning before now
10:28 and it may be our best option still
10:29 <rangermauve> RangerMauve Seeding is also pretty popular due to BT
10:29 <@pfrazee> Paul "Seeding service" is pretty darn clear if you know BT
10:29 "You need a seed"
10:29 <rangermauve> RangerMauve From the poll I ran on twitter it seems those two were pretty close. https://twitter.com/RangerMauve/status/1110604165233954816?s=20
10:29 <@pfrazee> Paul oh nice thanks
10:30 <rangermauve> RangerMauve I don't think that either of them are intuitive unless you already know what those words mean in the context of P2P. :P
10:30 <@pfrazee> Paul Emilia's point about the connotations of BitTorrent is pretty accurate
10:30 rangermauve: agree
10:31 "Durability" is an interesting theme to play with
10:33 <rangermauve> RangerMauve Durability seems good, but I think it would also be aimed more towards sysadmins and distributed systems people. I think that outside those contexts it implies that it keeps your Dat from snapping in half somehow
10:33 <@pfrazee> Paul good point...
10:33 okay maybe there's an "Uptime" or "Availability" theme?
10:33 <rangermauve> RangerMauve Same with mirroring, since that's more for people used to file sharing websites that have that
10:33 I think those have the same audience. :P
10:34 Which is OK, IMO
10:34 <@pfrazee> Paul well with uptime you can get an intuitive sensibility for some people, like, it's your remote presence
10:34 the positive about "mirror" is that it suggests it's a dumb thing that's just being a stand-in for you
10:35 <rangermauve> RangerMauve Yeah, I think mirror is also a pretty common word used for something like this.
10:35 <@pfrazee> Paul it's in the "not bad" category for me
10:35 <rangermauve> RangerMauve People thinking of Dat in the context of file sharing would already know what it means to some extent.
10:36 <@pfrazee> Paul it's also got history with websites
10:36 <rangermauve> RangerMauve Yeah!
10:36 Brb, feeding cats
10:36 <@pfrazee> Paul kk
10:37 another idea: "Keeper"
10:39 another idea: "Steward"
10:39 <@pfrazee> Paul another idea: "Base"
10:45 → mafintosh joined (sid28509@gateway/web/irccloud.com/x-dupfcvhgfkzoouca)
10:49 <rangermauve> RangerMauve I just have this nightmare where Amazon gets into P2P hosting and starts extending things with closed source functionality
10:49 Or some other big company
10:49 <@pfrazee> Paul well IMO, if dat becomes a thing, you should count on Amazon and Google and Microsoft doing that
10:50 pinning services are a hosting resource and they have hosting resources
10:51 <rangermauve> RangerMauve I guess they'll _have_ to add proprietary extensions if they want to differentiate themselves
10:51 <@pfrazee> Paul perhaps. I get that anxiety. It's literally what google is doing right now with AMP on the Web and Email
10:51 <mafintosh> Mathias Buus Hello
10:51 <@pfrazee> Paul mafintosh: hey
10:52 <rangermauve> RangerMauve 👋 Thinking of words that are more intuitive than pinning to people that don't already know what pinning is
10:52 I'm a fan of mirroring and seeding, but those still require you to have a certain background
10:53 <@pfrazee> Paul rangermauve: if we were to focus on there being lots of pinning (or X) services, what I'd feel the need to solve within beaker is a toolset for people to choose providers
10:54 so what I'd want to do is setup arrangements with providers to maintain an SLA that promises they'll stick around in exchange for being recommended by beaker
10:54 <rangermauve> RangerMauve Yeah. Part of the reason I really wanted to get this pinning stuff in the CLI (and the discovery-swarm-web stuff before) was to make the standards easy to use. There's a small bunch of companies working with Dat content, and I think they're all inventing their own pinning services and gateways
10:55 If we had a way to tell them "Here's what a pinning service should look like" then we'd have less fragmentation and something like that would be more feasible
10:55 <@pfrazee> Paul rangermauve: agree, and Im really glad you're doing that. Also, to be clear, Beaker will always make custom providers easy to setup too.
10:56 yes
11:01 <@pfrazee> Paul so, we can circle back to the "many providers vs a few" topic in the future, but from my perspective with beaker, what I need is reliable partners
11:02 <rangermauve> RangerMauve Agreed
11:02 <@pfrazee> Paul I'm going to have beaker include one-click default providers to signup for. I'd be 1000% happy if beaker didnt run them, and if there was more than one
11:04 but pinning providers should *not* have a way to differentiate from each other, and theyre incredibly low margin
11:04 so- let's setup another conversation about that and get darius involved
11:04 <rangermauve> RangerMauve Sounds good. :P
11:05 So all that to say, I think mirroring is a good term because there's usually lots of mirrors for a given file and it's a known term for people that do file sharing but maybe not p2p
11:06 <@pfrazee> Paul yeah, mirroring isnt bad. Let's try a few more terms
11:06 mafintosh: just to check, do you have any bikeshed suggestions on that?
11:07 <mafintosh> Mathias Buus are there any other things other than ipfs that use the word pinning today?
11:07 <@pfrazee> Paul no
11:07 <rangermauve> RangerMauve Should we use an existing term or invent a new one?
11:08 <@pfrazee> Paul I think both should be on the table
11:08 <mafintosh> Mathias Buus mirror is good
11:08 <@pfrazee> Paul (I'm just going to throw out random ideas that come into my head as we talk)
11:08 idea: Courier/carrier
11:09 <rangermauve> RangerMauve I'm personally more into using existing terms since it'll result in fewer people to educate. I'm usually pretty skeptical of P2P/Blockchain projects that invent a bunch of new terms for existing concepts
11:09 <@pfrazee> Paul rangermauve: oh yeah I hear that. 
11:10 <rangermauve> RangerMauve There was one I read recently, and it had like 20 terms and most of them were just existing concepts that got turned into marketing material
11:10 <mafintosh> Mathias Buus pfrazee: courier is no good for non native speakers
11:10 carrier also, they'd think aircraft carriers
11:10 <@pfrazee> Paul mafintosh: yeah. (No need to respond to any of my ideas if they dont land, Im just throwing em out)
11:11 <mafintosh> Mathias Buus :)
11:11 uptime service
11:11 <@pfrazee> Paul I was toying with "uptime" earlier
11:14 <rangermauve> RangerMauve Yeah, so it'd be cool to ~steal~, I mean _borrow_ existing terms
11:14 (I mean, that's why I think it would be cool)
11:15 <@pfrazee> Paul yeah
11:16 <mafintosh> Mathias Buus what are some existing terms? other than mirror
11:16 and hosting
11:16 <@pfrazee> Paul server
11:16 cloud
11:16 <rangermauve> RangerMauve seed, and pin? :P
11:16 <mafintosh> Mathias Buus replica
11:17 <@pfrazee> Paul replica...
11:18 <rangermauve> RangerMauve I think replica is a bit too techy. I don't think everyone knows about databases
11:18 <@pfrazee> Paul node
11:18 <mafintosh> Mathias Buus copy
11:18 <rangermauve> RangerMauve file share?
11:18 <@pfrazee> Paul store
11:18 <mafintosh> Mathias Buus i like store
11:19 <rangermauve> RangerMauve repository (that's what's in the CLI now)
11:19 <mafintosh> Mathias Buus 'can you store this dat for me'
11:19 <@pfrazee> Paul 'store service'
11:19 <rangermauve> RangerMauve storage service?
11:19 <mafintosh> Mathias Buus store sounds friendly to me
11:20 it's the kinda thing you'd say to your neighbor
11:20 <@pfrazee> Paul `dat store add`
11:20 <rangermauve> RangerMauve `dat store set http://hashbase.io` `dat store dat://foobar`
11:20 <@pfrazee> Paul that might have legs
11:20 <rangermauve> RangerMauve I like it :D
11:21 <@pfrazee> Paul okay let's sit on that for a bit
11:21 rangermauve: if you can just hold off merging your pinning tools into the CLI till we're sure
11:21 <rangermauve> RangerMauve Would you say "Store your dat in a store" or "Store your dat in a storage service" or "storage provider"?
11:21 Yeah, I've still got some work to do around configuration
11:22 Which is what I'm doing now
11:22 <@pfrazee> Paul I think store and storage can be somewhat interchangeable
11:23 but I'd call the service a "dat store" maybe?
11:23 I think this is pretty good
11:23 <rangermauve> RangerMauve What about when there's a marketplace of Dat apps? :P
11:23 I guess it'd be external to the protocol itself
11:24 <@pfrazee> Paul rangermauve: my thinking with beaker is that services and apps will be two different marketplaces with their own sets of sub-types
11:24 "dat stores" being one kind of service
11:24 and beaker will compose them for users
11:24 <rangermauve> RangerMauve Sounds good to me
11:29 <rangermauve> RangerMauve I'll get started on renaming the commands and stuff
11:29 <jhand> Joe Hand cloud friend
11:31 <jhand> Joe Hand can we make the command blue though
11:31 <rangermauve> RangerMauve :P
11:31 ??
11:31 <@pfrazee> Paul jhand: I prefer chartreuse
11:31 rangermauve: bikeshedding jokes
11:32 <rangermauve> RangerMauve Ha ha, that totally went over my head. 😂
11:32 <@pfrazee> Paul okay thanks yall for indulging me on this
11:32 <jhand> Joe Hand I like store!
11:32 <@pfrazee> Paul yeah me too :D
11:32 <rangermauve> RangerMauve Yeah, it's pretty nice
11:32 <@pfrazee> Paul I'll keep everyone updated on the meeting with darius, jhand I'll invite you as well
11:32 <jhand> Joe Hand though that gives us less flexibility around "registry" type pinning services

Mildly redacted to stay on topic

Conclusion:

We'll rename dat-pin to dat-store and use dat store dat://somekey along with saying "Put your read key in your dat store".

@pfrazee Maybe you could update the wording on hashbase to align with this term?

[pfrazee]

@RangerMauve Yeah I'll have to get that on my todos