Encountering "KeyError: 'rest_Id'" while using TwitterUserProfileScraper

[ronnypollak]

Hello!

I am using the TwitterUserProfileScraper to fetch the tweets and retweets of certain users. It works fine until I encounter the error KeyError: 'rest_id'.

I am not sure what causes this error. Can anybody help me with this?

The code I am using is the following:

for handle in afd_handles:
    for tweet in enumerate(
            sntwitter.TwitterProfileScraper(handle).get_items()):
        if date_start < tweet[1].date < date_end:
            tweets_list2.append(tweet[1])

The error message:

[JustAnotherArchivist]

What's your snscrape version, and do you have some examples of profiles that trigger this?

[ronnypollak]

Hello, my snscrape version is 0.4.3.20220106. For example, the user handle "JuergenBraunAfD" triggers this Error.

[JustAnotherArchivist]

Tweet 1567449406835724290 specifically causes the crash. It quotes 1567414879723610112, which by itself does not cause any issues, but when retrieved through the former, 1567276089122213889 (quoted in 1567414879723610112) is referenced under a different key.

[ronnypollak]

@JustAnotherArchivist With other users/tweets I also very frequently get the KeyError 'result'. For example with the user DrMEspendiller.

[JustAnotherArchivist]

@ronnypollak result? I see the same error as above on that one, KeyError: 'rest_id'. 1586039847688572928 is the tweet causing the crash.

[ronnypollak]

@JustAnotherArchivist I couldn't replicate it with the handle I sent before. But I can replicate it with the handle PetrBystronAfD

[ronnypollak]

@JustAnotherArchivist Another Error I can replicate is the KeyError: 'legacy with the user handle Renner_AFD for example.

[JustAnotherArchivist]

Yes, I can reproduce it with those. Please file separate issues since they're different bugs than the rest_id one.

[JustAnotherArchivist]

I'm unable to reproduce this crash anymore. I did get another KeyError: 'rest_id' crash, which is now fixed (564a5eca), but that's unrelated and comes from a tweet's card.

If anyone has another example of this crash on the quotedRefResult line, please let me know.

[JustAnotherArchivist]

@davira reported a user in #740 that seems to crash like this: IdiazAyuso. It's a TweetWithVisibilityResults object on tweet 1247569871782756354 (referencing 1247316200029380610). I'll treat that as the original issue here.