sfm-ui

sfm-ui copied to clipboard

To Do

• See Laura's commits: https://github.com/gwu-libraries/sfm-ui/tree/twitter-v2
• Because of the tiered access to the v. 2 API, users need to see error messages/in-app documentation related to the following:
  • Using (streaming) operators they are not permitted to use
  • Trying to add more stream rules than they are allowed (not sure if this triggers an error or just replaces the rules currently in use)
  • Exceeding the monthly limit
  • Trying to use the search_all endpoint without Academic Research access
• Users can select which fields they want to return, including annotations. We need to provide a mechanism for users to do this.

Streaming API

• Add a way for users to review their current streaming rules (since they're limited by account type to either 5, 25, or 1,000).
• Add support for new query operators in creating stream rules.

Search API

• Test Laura's editions

Note that streaming rules do not persist indefinitely. After 180 days of inactivity, Twitter deletes them. (The email address on the associated developer account will receive messages at 150 and 180 days, notifying the user of this action.) So we may want to build in a check (if the date of the last harvest > 180 days, assume the streaming rules need to be resent).

Notes from 12/22/22

• To investigate: When a user submits a new streaming rule, does the harvest need to be restarted?
• May need to keep track of the user's streaming rules internally for the purposes of the manifest.
• Let's handle rules in the same way we're handling seeds.
  • New logic: API call to add/delete a stream rule (seed) and to handle possible errors
  • Users are responsible for ensuring they are running a single filter-stream harvest at a time – can use existing logic to handle this.

Twitter filter-stream-rule operators

• Maybe present two options in the UI:
  1. Free text box for rule construction
  2. A few structured field entries for the most common parameters
• Model would need an indicator as to which kind of form the user has populated ("Advanced" or not)