snscrape icon indicating copy to clipboard operation
snscrape copied to clipboard

Crash on `TweetUnavailable` results

Open TheTechRobo opened this issue 2 years ago • 3 comments

snscrape twitter-tweet --recurse 1500507885922619394
Traceback (most recent call last):
  File "/home/thetechrobo/.local/bin/snscrape", line 8, in <module>
    sys.exit(main())
  File "/home/thetechrobo/.local/lib/python3.9/site-packages/snscrape/_cli.py", line 308, in main
    for i, item in enumerate(scraper.get_items(), start = 1):
  File "/home/thetechrobo/.local/lib/python3.9/site-packages/snscrape/modules/twitter.py", line 1030, in get_items
    for tweet in self._graphql_timeline_instructions_to_tweets(obj['data']['threaded_conversation_with_injections']['instructions'], includeConversationThreads = True):
  File "/home/thetechrobo/.local/lib/python3.9/site-packages/snscrape/modules/twitter.py", line 688, in _graphql_timeline_instructions_to_tweets
    yield self._graphql_timeline_tweet_item_result_to_tweet(item['item']['itemContent']['tweet_results']['result'])
  File "/home/thetechrobo/.local/lib/python3.9/site-packages/snscrape/modules/twitter.py", line 652, in _graphql_timeline_tweet_item_result_to_tweet
    raise snscrape.base.ScraperException(f'Unknown result type {result["__typename"]!r}')
snscrape.base.ScraperException: Unknown result type 'TweetUnavailable'

Dump: https://transfer.archivete.am/60upN/stack

TheTechRobo avatar Mar 24 '22 20:03 TheTechRobo

I've seen this before during my card code tests, but I have no idea yet what causes it. The web interface simply doesn't show the requested tweet at all in these cases. It seems somewhat related to tombstones (#392) per Twitter's JS, but the returned data has zero context details, unlike proper tombstones.

Two of the IDs causing it in this particular example: 1500514518631387140 1500529138691710987

JustAnotherArchivist avatar Mar 25 '22 05:03 JustAnotherArchivist

At least those two examples are showing as age-restricted now (cf. #419).

JustAnotherArchivist avatar Apr 12 '22 19:04 JustAnotherArchivist

Yeah, I noticed that too earlier today.

TheTechRobo avatar Apr 12 '22 19:04 TheTechRobo

I think I found a reliable way of reproducing this while implementing communities (#614): 18+ posts in communities appear to always be returned as TweetUnavailable. Community 1560462400503296001 has one of those.

JustAnotherArchivist avatar Feb 21 '23 19:02 JustAnotherArchivist