lmatteis
Followers should be a separate linked list
First of all, awesome ideas!
So, when you follow someone, it is published as part of the tweets linked list, this feels odd for a couple reasons:
- If you follow someone, and then publish a lot of tweets, you have to crawl a lot until you can discover who you follow to see their latest tweets
- When somebody else is crawling your tweets to read in their timelines, they don't care about who you follow, so it's unnecessary steps
My suggestion would be having two linked lists on the head instead of one: the tweets list and the following list.
The DhtDownload code branches out followers tweets anyway, it should even make this code simpler.
Oh, I didn't knew Dht Network only lasted about 30 minutes without republishing, so we must republish others following activity as well, right?
Having a separate list makes sense. It does make things simpler when republishing because currently I'm recursively branching out when I find a follower. However the head might start to get bloated. Perhaps there can just be a single 20 byte field (not up to 80 like the regular next fields) and avoid doing the skip list stuff.
Yes. DhtDownload already downloads all your followers tweets and DhtPublish republishes them every 30 minutes.
Good idea :+1:
This way we could limit the tweets list republication, reducing the work to be done by the clients as the number os tweets grows. Eventually, very old and unpopular tweets would stop being published, but I don't think that would be much of a problem :D
As for the followers list, we can't lose that
I was thinking of limiting the tweet list republication as well. Any ideas? Nobody really browses through old tweets anyway and like you said it would reduce the work needed by followers, and it would certainly reduce the load on the DHT (which is already quite brittle).
My initial thought was that followers would only store the "latest 20 tweets" or something along those lines. So in that sense the list would be more like a FIFO queue. But I'm open to other thoughts.
So I've thought a bit about this and currently I'm not going to implement it because the idea of a single list is more elegant in my opinion. Also implementing another list requires some drastic changes to the code and I don't feel like breaking stuff.
You're right that getting the whole list of followers requires an expensive crawl, but once you follow 200-300 people, also crawling the "followers list" becomes expensive. I'm thinking that users will share their followers list in some other way. For instance, imagine every month posting a gist url of your current followers. Less bytes on the DHT.
Perhaps in the future if we see this is really needed, I'll make an effort.
Alright, no problem. My only concern is that this kind of stuff gets harder to change as it becomes popular because it means to introduce breaking changes in the network itself, which is hard to migrate.
And I really want to see this getting in the main stream!
About the followers list, since we don't need to put tweets text there, we could put like 10 hashes per block before linking to the next followers list block. So for 300 people you'd load 30 requests.
I'm not saying that you have to do it, it is your call, but anyone reading this might suggest a PR :)