social-feed-manager
social-feed-manager copied to clipboard
gwu-libraries
"Old SFM" -- manage rules and streams from social data sources, starting with twitter.
Suggested by https://twitter.com/mart1nkle1n - UCLA is working around this by adding Twarc, so clearly there is a need. Use case is a news event occurs; we notice a few hours/days...
Adding a lengthy (over around 100 users) userlist will tend to hit rate limits with the current implementation. Ref #126, #283 where bulk lookups were implemented for update_usernames.
On a large sfm instance (millions of TwitterUserItems, in our case >10m) the max(twitter_id) lookup in user_timeline is very slow. It filters on twitter_user_id, so a composite index on twitter_user_id...
The user_timeline job becomes inefficient when there are many TwitterUsers whose accounts are no longer active in Twitter. I would recommend adding a --prune flag to the update_usernames management command,...
...and therefore, when supervisor is started, the filterstream either: - in the case of an update to the parameters, won't reflect the updated parameters - in the case of enable/create,...
Needs to point to http://social-feed-manager.readthedocs.org/en/latest/data_dictionary.html
Probably need at least a try/catch block around file operations.
It seems that there is not an error from Tweepy we're expecting anymore-- it was originally watching for an error that was not going to occur with statuses_lookup. If there...
fetch_tweets_by_id should be logging ids that were not returned by statuses_lookup, but it is not. The current fetch() method appears to assume that statuses_lookup throws and exception when any of...
Results from hydration in fetch_tweets_by_id are getting an extra newline between tweets. We removed them from filterstream results (see #321). This could be easily changed.
Metadata
Owner
Metadata
"Old SFM" -- manage rules and streams from social data sources, starting with twitter.