Scraper fails with 400 Bad Request: Missing features

[DosAmp]

I find myself unable to start scraping in online mode:

[info]    Server started on: http://localhost:4788
[error]   twitter: failed to fetch response body: 400 Bad Request

The Bookmarks API request sent by my browser currently looks like this after parsing:

{
   "variables": {
      "count": 20,
      "includePromotedContent": true
   },
   "features": {
      "graphql_timeline_v2_bookmark_timeline": true,
      "rweb_lists_timeline_redesign_enabled": true,
      "responsive_web_graphql_exclude_directive_enabled": true,
      "verified_phone_label_enabled": false,
      "creator_subscriptions_tweet_preview_api_enabled": true,
      "responsive_web_graphql_timeline_navigation_enabled": true,
      "responsive_web_graphql_skip_user_profile_image_extensions_enabled": false,
      "tweetypie_unmention_optimization_enabled": true,
      "responsive_web_edit_tweet_api_enabled": true,
      "graphql_is_translatable_rweb_tweet_is_translatable_enabled": true,
      "view_counts_everywhere_api_enabled": true,
      "longform_notetweets_consumption_enabled": true,
      "responsive_web_twitter_article_tweet_consumption_enabled": false,
      "tweet_awards_web_tipping_enabled": false,
      "freedom_of_speech_not_reach_fetch_enabled": true,
      "standardized_nudges_misinfo": true,
      "tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled": true,
      "longform_notetweets_rich_text_read_enabled": true,
      "longform_notetweets_inline_media_enabled": true,
      "responsive_web_media_download_video_enabled": false,
      "responsive_web_enhance_cards_enabled": false
   },
   "fieldToggles": {
      "withArticleRichContentState": false
   }
}

When I include the current feature set of Scraper.features in the URL, I get the following error using curl:

HTTP/2 400
{"errors":[{"message":"The following features cannot be null: responsive_web_twitter_article_tweet_consumption_enabled, responsive_web_media_download_video_enabled","extensions":{"name":"BadRequestError","source":"Client","code":336,"kind":"Validation","tracing":{"trace_id":"xxxxxxxxxxxxxxxx"}},"code":336,"kind":"Validation","name":"BadRequestError","source":"Client","tracing":{"trace_id":"xxxxxxxxxxxxxxxx"}}]}

After adding at least these two features, the request succeeds. Therefore, I conclude that this change should only require an update to the API feature set. The fieldToggles query parameter still does not seem to be needed at this time.

[Webklex]

Hi @DosAmp , thanks for reporting this issue and taking the time to look into it. It turns out, there has been an even bigger change; they removed a previously used endpoint https://twitter.com/i/api/2/timeline/conversation/{ID}.json.

This means that the "new" graph api has to be used from now on.. However this is a bit easier said than done and requires a bit of work..

If you like to poke around, please feel welcome to do so. I've fixed what I could and added a forced exit together with some comments to the problematic section.

Thanks again :)

[DosAmp]

Thank you. Without the forced exit, I can at least backup my bookmarks in a machine-readable format. They lack thread information, but I've been bookmarking mostly standalone tweets anyway, and I wrote a little Python script to extract and download media attached to them for now.

The advantage of the new GraphQL response is that it's similar to the existing BookmarkResponse, except that it contains not only an entry of type TimelineTimelineItem, which acts as a wrapper around a TimelineTweet of the tweet in question, but also an entry of type TimelineTimelineModule containing the other tweets in the conversation.

[dooooken]

FYI it does appear that TBM just works now, unsure of when it started working again. But it's pulling and saving bookmarks as expected.