discursive icon indicating copy to clipboard operation
discursive copied to clipboard

Published 20 hours ago • verified publisher icon Data4Democracy

Poll account data daily

Open bstarling opened this issue 8 years ago • 4 comments

Process that will accept a list of users and poll daily. Should save a time stamp and user attributes :

  • followers
  • bio
  • Location
  • Profile picture/Banner?

Allows us to see how a profile changes over time, fastest growing profiles etc.

bstarling avatar Jan 15 '17 14:01 bstarling

I can help work on this one this week (if anyone wants to join, let's do it!). It's a key part of the whole Twitter analytics piece. Aim for a review-ready PR by COB Friday (20 JAN 2017).

hadoopjax avatar Jan 15 '17 17:01 hadoopjax

Since this could be used by any number of teams thoughts on using this as a pilot project for starting to add twitter functionality tocollect-social? See related.

bstarling avatar Jan 17 '17 19:01 bstarling

Hi, is this issue still open to being worked on? If yes, I would like to contribute :-). I already got a testing-instance of discursive running on my amazon aws account and have some thoughts on how to tackle this in accordance to the already existing code.

  • I would use the lookup_users method of tweepy to pull the searched users, given in a text-file, off the REST-API.
  • I don't think a stream is needed for this like it is for the tweets.
  • To add the timing I would add another crontab-job.
  • The data would be stored in a new ElasticSearch-Index for users.
  • Create a time stamp by getting the system time at one point and add it to the data.

tdraebing avatar Feb 28 '17 09:02 tdraebing

just left a note in the PR @tdraebing but I'll test locally and get back to you. thanks for the work this is awesome!

hadoopjax avatar Mar 01 '17 01:03 hadoopjax