Consider adding use of Twitter search API

[kerchner]

Suggested by https://twitter.com/mart1nkle1n - UCLA is working around this by adding Twarc, so clearly there is a need. Use case is a news event occurs; we notice a few hours/days later and enable a new TwitterFilter, but missed the initial "spike". The missed data is recent enough that the Twitter search API would likely provide most of it.

[martinklein0815]

+1 thus far I have started an SFM filter to collect tweets moving forward and a twarc instance for tweets from the last 7-9 days (whatever the API provides). To me, this is a) one step too many and b) requires painful deduping.

[dchud]

Thanks @martinklein0815 - we will likely integrate search later this year. In the meantime you might get some value out of the since_id and max_id parameters to ensure the twarc searches don't overlap in time with streamed data. Of course there can be holes in the streamed data as well, but the stream gaps should also be fillable with these params tightening the search.

My apologies if you've already worked all this in to your plan, the main point is well taken!

[martinklein0815]

Thanks Dan, glad to hear! I have been using native Twitter API params 'since' and 'until' in the past with datetimes but params/values on the tweet id level might be more reliable - will test it out. Thanks for the pointer!

[kerchner]

@martinklein0815 Just keeping you in the loop, in the "new" SFM project ( https://github.com/gwu-libraries/sfm-ui ) we have built in use of the Twitter search API. We have a stable version out already, but if you can hang on for just a few weeks, we are close to releasing a 1.0 version.