snscrape icon indicating copy to clipboard operation
snscrape copied to clipboard
verified publisher icon JustAnotherArchivist

Unable to use proxy with snscrape

Open DrSocket opened this issue 3 years ago • 18 comments

Describe the bug

I'm getting a ScraperException when trying to use a proxy with snscrape on twitter.

I've tested the proxy ip is not banned by twitter by adding it to network configuration as well as just the browser. checked the browser is using the proxy ip on https://httpbin.org/ip.

Used a private browser window to connect to twitter.com and this worked normally. So it doesn't seem to be a proxy issue or a twitter blocking issue. I think it's something to do with how the python requests library is configuring the cipher when sending the request through proxy but I don't really know.

How to reproduce

Example of code reproducing the error. the proxy should work on twitter.com in a private browser window without being logged in.

tweets = sntwitter.TwitterSearchScraper('ukraine', proxies = {'https': 'http://10.10.10.10:10100'}).get_items()
t_list = []
for i, t in enumerate(tweets):
	if i >= 10:
		break
	t_list.append(t)

Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: non-200 status code (404)
4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.
Errors: blocked (403), non-200 status code (404), non-200 status code (404), non-200 status code (404)

Expected behaviour

The same code without proxy outputs correct response.

tweets = sntwitter.TwitterSearchScraper('ukraine').get_items()
t_list = []
for i, t in enumerate(tweets):
	if i >= 10:
		break
	t_list.append(t)

print (t_list)

[Tweet(url='https://twitter.com/Handoftheking00/status/1643024605844586497', date=datetime.datetime(2023, 4, 3, 22, 56, 6, tzinfo=datetime.timezone.utc), rawContent='@Spriter99880 Ukraine used world war 1 guns to sink the Moskva? Shame to Russia then https://t.co/WpzLX8S2kb', renderedContent='@Spriter99880 Ukraine used world war 1 guns to sink the Moskva? Shame to Russia then https://t.co/WpzLX8S2kb', id=1643024605844586497, user=User(username='Handoftheking00', id=1481954810471624704, displayname="Peter Obi's Last Daughter", rawDescription='Labour Party', renderedDescription='Labour Party', descriptionLinks=None, verified=False, created=datetime.datetime(2022, 1, 14, 11, 43, 7, tzinfo=datetime.timezone.utc), followersCount=736, friendsCount=248, statusesCount=52874, favouritesCount=68069, listedCount=0, mediaCount=97, location='Lagos, Nigeria', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1633043916227321856/Ya3498BU_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/1481954810471624704/1664374704', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643007181082509314, lang='en', source='<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>', sourceUrl='http://twitter.com/download/android', sourceLabel='Twitter for Android', links=None, media=[Photo(previewUrl='https://pbs.twimg.com/media/Fs0yK9PXgAMx1-N?format=jpg&name=small', fullUrl='https://pbs.twimg.com/media/Fs0yK9PXgAMx1-N?format=jpg&name=orig', altText=None)], retweetedTweet=None, quotedTweet=None, inReplyToTweetId=1643007181082509314, inReplyToUser=User(username='Spriter99880', id=1128337957289697281, displayname='Spriter', rawDescription=None, renderedDescription=None, descriptionLinks=None, verified=None, created=None, followersCount=None, friendsCount=None, statusesCount=None, favouritesCount=None, listedCount=None, mediaCount=None, location=None, protected=None, link=None, profileImageUrl=None, profileBannerUrl=None, label=None), mentionedUsers=[User(username='Spriter99880', id=1128337957289697281, displayname='Spriter', rawDescription=None, renderedDescription=None, descriptionLinks=None, verified=None, created=None, followersCount=None, friendsCount=None, statusesCount=None, favouritesCount=None, listedCount=None, mediaCount=None, location=None, protected=None, link=None, profileImageUrl=None, profileBannerUrl=None, label=None)], coordinates=None, place=None, hashtags=None, cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/JeffAdams82/status/1643024601457229826', date=datetime.datetime(2023, 4, 3, 22, 56, 5, tzinfo=datetime.timezone.utc), rawContent="So @AsaHutchinson is full on-board with continuing to funnel unlimited $$ into Ukraine. \nHe's a uni-party neocon globalist that no one outside of his own family would vote for.\nGo away, Doofus", renderedContent="So @AsaHutchinson is full on-board with continuing to funnel unlimited $$ into Ukraine. \nHe's a uni-party neocon globalist that no one outside of his own family would vote for.\nGo away, Doofus", id=1643024601457229826, user=User(username='JeffAdams82', id=35725414, displayname='Jeff Adams', rawDescription='John 3:16. Baltimore born, Os and Ravens fan; lover of classic tv, Aurora AFX slot cars, big news junky, reader of all things, dog, cat and wild critter fan', renderedDescription='John 3:16. Baltimore born, Os and Ravens fan; lover of classic tv, Aurora AFX slot cars, big news junky, reader of all things, dog, cat and wild critter fan', descriptionLinks=None, verified=False, created=datetime.datetime(2009, 4, 27, 12, 6, 16, tzinfo=datetime.timezone.utc), followersCount=734, friendsCount=829, statusesCount=35418, favouritesCount=27878, listedCount=16, mediaCount=3294, location='', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/2600717412/08isxhk7iseyef8aockg_normal.jpeg', profileBannerUrl='https://pbs.twimg.com/profile_banners/35725414/1647877264', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024601457229826, lang='en', source='<a href="http://twitter.com/download/android" rel="nofollow">Twitter for Android</a>', sourceUrl='http://twitter.com/download/android', sourceLabel='Twitter for Android', links=None, media=None, retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=[User(username='AsaHutchinson', id=269992801, displayname='Gov. Asa Hutchinson', rawDescription=None, renderedDescription=None, descriptionLinks=None, verified=None, created=None, followersCount=None, friendsCount=None, statusesCount=None, favouritesCount=None, listedCount=None, mediaCount=None, location=None, protected=None, link=None, profileImageUrl=None, profileBannerUrl=None, label=None)], coordinates=None, place=None, hashtags=None, cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/Dad_of_Marine/status/1643024600622858241', date=datetime.datetime(2023, 4, 3, 22, 56, 5, tzinfo=datetime.timezone.utc), rawContent='US ZIO Regime is the Biggest Supporter of  "TERRORISM"  anywhere &amp; at any time on this planet. With our hard earned tax $$, they fund Israel, Biggest "TERRORIST" org. ever. US funds Ukraine NAZI Terrorists, ISIS, al Qaeda in Syria, Boko Haram in Africa, Contras in Nicaragua, etc.', renderedContent='US ZIO Regime is the Biggest Supporter of  "TERRORISM"  anywhere &amp; at any time on this planet. With our hard earned tax $$, they fund Israel, Biggest "TERRORIST" org. ever. US funds Ukraine NAZI Terrorists, ISIS, al Qaeda in Syria, Boko Haram in Africa, Contras in Nicaragua, etc.', id=1643024600622858241, user=User(username='Dad_of_Marine', id=950710129, displayname='Dad_of_Marine', rawDescription='Born & Bred Ca.Mestizo-ArmyVet,MarineSon-DespiseZios-WeCannotAllowCabalOfZioPsychopathstoControlOurLives;PowerConcedesNothingWithoutDemand.StandwithOppressed', renderedDescription='Born & Bred Ca.Mestizo-ArmyVet,MarineSon-DespiseZios-WeCannotAllowCabalOfZioPsychopathstoControlOurLives;PowerConcedesNothingWithoutDemand.StandwithOppressed', descriptionLinks=None, verified=False, created=datetime.datetime(2012, 11, 16, 0, 25, 27, tzinfo=datetime.timezone.utc), followersCount=801, friendsCount=601, statusesCount=46997, favouritesCount=29031, listedCount=5, mediaCount=2411, location='', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1441969118802972677/Eaeobq_f_normal.jpg', profileBannerUrl=None, label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024600622858241, lang='en', source='<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App</a>', sourceUrl='https://mobile.twitter.com', sourceLabel='Twitter Web App', links=None, media=None, retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=None, cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/Bikers4Ukraine/status/1643024600442216456', date=datetime.datetime(2023, 4, 3, 22, 56, 5, tzinfo=datetime.timezone.utc), rawContent='Odesa from my window just now. This catches only the tail end of the volley. My 30 story building shook hard. Another good reason to own a helmet while volunteering in Ukraine. https://t.co/RgJAWMyu7b', renderedContent='Odesa from my window just now. This catches only the tail end of the volley. My 30 story building shook hard. Another good reason to own a helmet while volunteering in Ukraine. https://t.co/RgJAWMyu7b', id=1643024600442216456, user=User(username='Bikers4Ukraine', id=1584701678254579712, displayname='Silent Cycle Squadron - Nonprofit File No. 5506922', rawDescription='SCS and UMGC have teamed up! Bringing the Ukraine Military Electric Motorcycles for Non-Combat Ops. 1st Deployed 1-17-23.', renderedDescription='SCS and UMGC have teamed up! Bringing the Ukraine Military Electric Motorcycles for Non-Combat Ops. 1st Deployed 1-17-23.', descriptionLinks=None, verified=False, created=datetime.datetime(2022, 10, 25, 0, 22, 23, tzinfo=datetime.timezone.utc), followersCount=2411, friendsCount=4935, statusesCount=4616, favouritesCount=4334, listedCount=22, mediaCount=304, location='Odesa, Ukraine ', protected=False, link=TextLink(text='paypal.com/pools/c/8SLE8u…', url='https://www.paypal.com/pools/c/8SLE8uMiNq', tcourl='https://t.co/mwmmiYZEsy', indices=(0, 23)), profileImageUrl='https://pbs.twimg.com/profile_images/1636379697562161152/L8TNBYO5_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/1584701678254579712/1666688466', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024600442216456, lang='en', source='<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', sourceUrl='http://twitter.com/download/iphone', sourceLabel='Twitter for iPhone', links=None, media=[Video(thumbnailUrl='https://pbs.twimg.com/ext_tw_video_thumb/1643024525167144962/pu/img/GQCeJWuySaqyIXKS.jpg', variants=[VideoVariant(url='https://video.twimg.com/ext_tw_video/1643024525167144962/pu/vid/480x852/iXEerQFlgGiZvL6o.mp4?tag=12', contentType='video/mp4', bitrate=950000), VideoVariant(url='https://video.twimg.com/ext_tw_video/1643024525167144962/pu/pl/tlmw5mTW7QIeV6U6.m3u8?tag=12&container=fmp4', contentType='application/x-mpegURL', bitrate=None), VideoVariant(url='https://video.twimg.com/ext_tw_video/1643024525167144962/pu/vid/320x568/Nrz28zrW-ylNKHTs.mp4?tag=12', contentType='video/mp4', bitrate=632000), VideoVariant(url='https://video.twimg.com/ext_tw_video/1643024525167144962/pu/vid/720x1280/2kIjRPjVZobCYv3Q.mp4?tag=12', contentType='video/mp4', bitrate=2176000)], duration=51.235, views=0, altText=None)], retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=None, cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/Brian_Pinelli/status/1643024599439876097', date=datetime.datetime(2023, 4, 3, 22, 56, 5, tzinfo=datetime.timezone.utc), rawContent='Polish president Duda: "In two days I\'m going to host in #Poland the premier Volodymyr Zelenskyy, the president of #Ukraine, who is coming to meet my compatriots in Poland, but also his compatriots living in Poland" https://t.co/6bHMKHBKkf', renderedContent='Polish president Duda: "In two days I\'m going to host in #Poland the premier Volodymyr Zelenskyy, the president of #Ukraine, who is coming to meet my compatriots in Poland, but also his compatriots living in Poland" https://t.co/6bHMKHBKkf', id=1643024599439876097, user=User(username='Brian_Pinelli', id=42775378, displayname='Brian Pinelli', rawDescription='Journalist. Spanning the globe 🌎 to bring you the constant variety of sport, the thrill of victory 🏆 and the agony of defeat. 9 Olympics. 🔥', renderedDescription='Journalist. Spanning the globe 🌎 to bring you the constant variety of sport, the thrill of victory 🏆 and the agony of defeat. 9 Olympics. 🔥', descriptionLinks=None, verified=False, created=datetime.datetime(2009, 5, 27, 1, 2, 52, tzinfo=datetime.timezone.utc), followersCount=1746, friendsCount=1601, statusesCount=12877, favouritesCount=21349, listedCount=131, mediaCount=3139, location='Rome, Lazio', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1357825338944991233/5P5P5tDo_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/42775378/1603496256', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024599439876097, lang='en', source='<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', sourceUrl='http://twitter.com/download/iphone', sourceLabel='Twitter for iPhone', links=None, media=[Photo(previewUrl='https://pbs.twimg.com/media/Fs0yKelWIAEd18p?format=jpg&name=small', fullUrl='https://pbs.twimg.com/media/Fs0yKelWIAEd18p?format=jpg&name=orig', altText=None)], retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=['Poland', 'Ukraine'], cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/socialismoolib1/status/1643024599079157761', date=datetime.datetime(2023, 4, 3, 22, 56, 4, tzinfo=datetime.timezone.utc), rawContent='🇺🇦#UKRAINE #US\n\n#News \n\n 🔫04/03/2023  Day 404\n\n#Zelensky: Heavily fought over #Bakhmut, "especially hot"\n\n#Russia has suffered up to 200,000 casualties in #UkraineWar by poor weapon handling drills, road traffic accidents, climatic injuries(hypothermia) \n\nhttps://t.co/zUpJC8gSuT', renderedContent='🇺🇦#UKRAINE #US\n\n#News \n\n 🔫04/03/2023  Day 404\n\n#Zelensky: Heavily fought over #Bakhmut, "especially hot"\n\n#Russia has suffered up to 200,000 casualties in #UkraineWar by poor weapon handling drills, road traffic accidents, climatic injuries(hypothermia) \n\ndailykos.com/stories/2023/4…', id=1643024599079157761, user=User(username='socialismoolib1', id=1372145833483862018, displayname='GLOBAL NEWS-NOTICIAS DEL MUNDO-NOUVELLES DU MONDE', rawDescription='SOCIALISM=COMMJNISM=DICTATORSHIP=POOR=WAR\n \nSOCIALISMO=COMUNISMO=DICTADURA=POBRE=GUERRA\n\nFREEDOM=DEMOCRACY=CAPITALISM=RICH\n\nLIBERTAD=DEMOCRACIA=CAPITALISMO=RICO', renderedDescription='SOCIALISM=COMMJNISM=DICTATORSHIP=POOR=WAR\n \nSOCIALISMO=COMUNISMO=DICTADURA=POBRE=GUERRA\n\nFREEDOM=DEMOCRACY=CAPITALISM=RICH\n\nLIBERTAD=DEMOCRACIA=CAPITALISMO=RICO', descriptionLinks=None, verified=False, created=datetime.datetime(2021, 3, 17, 11, 25, 55, tzinfo=datetime.timezone.utc), followersCount=2752, friendsCount=5038, statusesCount=6259, favouritesCount=3, listedCount=2, mediaCount=2046, location='NEWS AGENCY-AGENCIA DE PRENSA ', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1415370981359640577/4o0hdTUy_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/1372145833483862018/1631992698', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024599079157761, lang='en', source='<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App</a>', sourceUrl='https://mobile.twitter.com', sourceLabel='Twitter Web App', links=[TextLink(text='dailykos.com/stories/2023/4…', url='https://www.dailykos.com/stories/2023/4/2/2161732/-Ukraine-Invasion-Day-404', tcourl='https://t.co/zUpJC8gSuT', indices=(256, 279))], media=None, retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=['UKRAINE', 'US', 'News', 'Zelensky', 'Bakhmut', 'Russia', 'UkraineWar'], cashtags=None, card=SummaryCard(title='Ukraine Invasion Day 404: military blogging can be dangerous', url='https://www.dailykos.com/stories/2023/4/2/2161732/-Ukraine-Invasion-Day-404', description='Probably easy to make enemies in Wagner much like accusing Erik Prince of wanting to be Viscount of Afghanistan. Prominent military blogger #VladlenTatarsky killed & 30 wounded in cafe in #StPeters...', thumbnailUrl='https://pbs.twimg.com/card_img/1642777721250906113/LYmxZZdc?format=jpg&name=orig', siteUser=User(username='dailykos', id=20818801, displayname='Daily Kos', rawDescription="Daily Kos is the nation's largest online progressive media and activism hub.", renderedDescription="Daily Kos is the nation's largest online progressive media and activism hub.", descriptionLinks=None, verified=True, created=datetime.datetime(2009, 2, 13, 23, 51, 25, tzinfo=datetime.timezone.utc), followersCount=277808, friendsCount=0, statusesCount=119455, favouritesCount=3006, listedCount=6954, mediaCount=3928, location='', protected=False, link=TextLink(text='dailykos.com', url='http://dailykos.com/', tcourl='https://t.co/DqzP3PIDdS', indices=(0, 23)), profileImageUrl='https://pbs.twimg.com/profile_images/1169343346067877890/SGSBWvdR_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/20818801/1567624996', label=None), creatorUser=User(username='annieli', id=16069334, displayname='annieli', rawDescription='', renderedDescription='', descriptionLinks=None, verified=False, created=datetime.datetime(2008, 8, 31, 18, 53, 51, tzinfo=datetime.timezone.utc), followersCount=28, friendsCount=1011, statusesCount=194, favouritesCount=16, listedCount=2, mediaCount=2, location='', protected=True, link=TextLink(text='dailykos.com/user/annieli', url='http://www.dailykos.com/user/annieli', tcourl='http://t.co/0SlKwo5Coc', indices=(0, 22)), profileImageUrl='https://pbs.twimg.com/profile_images/2940690061/40f1dc31f58919f8c51269472df8b9b5_normal.jpeg', profileBannerUrl=None, label=None)), viewCount=None, vibe=None), Tweet(url='https://twitter.com/jmuhj1/status/1643024579009593344', date=datetime.datetime(2023, 4, 3, 22, 56, tzinfo=datetime.timezone.utc), rawContent='🥰🆘❤️🐈❤️🆘📢#AdoptCat #AdoptAShelterCat #AdoptDontShop #AdoptDontBuy #cats #kittens #SaveThemAll #spay #neuter #TNR AdoptFosterDonateShare for felines needing loving homes. NOW more than ever THEY NEED US &amp; WE NEED THEM-UNCONDITIONAL LOYALTY &amp; LOVE 24/7-365!🥰🆘🆘🆘🆘🆘🆘🆘🆘🆘', renderedContent='🥰🆘❤️🐈❤️🆘📢#AdoptCat #AdoptAShelterCat #AdoptDontShop #AdoptDontBuy #cats #kittens #SaveThemAll #spay #neuter #TNR AdoptFosterDonateShare for felines needing loving homes. NOW more than ever THEY NEED US &amp; WE NEED THEM-UNCONDITIONAL LOYALTY &amp; LOVE 24/7-365!🥰🆘🆘🆘🆘🆘🆘🆘🆘🆘', id=1643024579009593344, user=User(username='jmuhj1', id=16086855, displayname='jmuhj1 STAND WITH UKRAINE & free world!', rawDescription='Cats are the light of my life. Love desert, spirit, nature.   World citizen. I collect cat & cat lover accounts 4 advocacy. Building bridges. Raspy aspy. NO DMs', renderedDescription='Cats are the light of my life. Love desert, spirit, nature.   World citizen. I collect cat & cat lover accounts 4 advocacy. Building bridges. Raspy aspy. NO DMs', descriptionLinks=None, verified=False, created=datetime.datetime(2008, 9, 1, 19, 42, 15, tzinfo=datetime.timezone.utc), followersCount=3524, friendsCount=4987, statusesCount=1182520, favouritesCount=440418, listedCount=168, mediaCount=6184, location='So Cali', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1497275583964737536/SqFn1Mrm_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/16086855/1645813944', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024579009593344, lang='en', source='<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App</a>', sourceUrl='https://mobile.twitter.com', sourceLabel='Twitter Web App', links=None, media=None, retweetedTweet=None, quotedTweet=Tweet(url='https://twitter.com/NYCDEATHROWCATS/status/1624152769106960390', date=datetime.datetime(2023, 2, 10, 21, 6, 10, tzinfo=datetime.timezone.utc), rawContent='***RETURN*** Tater Tot was originally adopted in 2018. He is now 10 years old and his owner has returned him to the shelter. Tater Tot is already fixed and very friendly. He is hoping someone will give him a new furever home this time. https://t.co/SFUQsP6NXI https://t.co/DYlMX5O0U2', renderedContent='***RETURN*** Tater Tot was originally adopted in 2018. He is now 10 years old and his owner has returned him to the shelter. Tater Tot is already fixed and very friendly. He is hoping someone will give him a new furever home this time. facebook.com/NYCDEATHROWCAT… https://t.co/DYlMX5O0U2', id=1624152769106960390, user=User(username='NYCDEATHROWCATS', id=548036878, displayname="NYC's DEATH ROW CATS", rawDescription='Welcome to NYCs Death Row Cats Page. This page was created to advocate for the cats at the NYC high kill shelters.', renderedDescription='Welcome to NYCs Death Row Cats Page. This page was created to advocate for the cats at the NYC high kill shelters.', descriptionLinks=None, verified=False, created=datetime.datetime(2012, 4, 8, 1, 17, 46, tzinfo=datetime.timezone.utc), followersCount=26083, friendsCount=51, statusesCount=122547, favouritesCount=20, listedCount=250, mediaCount=64530, location='', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1440114068954755072/bnOFtZCT_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/548036878/1533421801', label=None), replyCount=14, retweetCount=274, likeCount=206, quoteCount=13, conversationId=1624152769106960390, lang='en', source='<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App</a>', sourceUrl='https://mobile.twitter.com', sourceLabel='Twitter Web App', links=[TextLink(text='facebook.com/NYCDEATHROWCAT…', url='https://www.facebook.com/NYCDEATHROWCATS/photos/a.2151686061509881/2209658359045984/', tcourl='https://t.co/SFUQsP6NXI', indices=(236, 259))], media=[Photo(previewUrl='https://pbs.twimg.com/media/FoomTa9WIAEuZ4e?format=jpg&name=small', fullUrl='https://pbs.twimg.com/media/FoomTa9WIAEuZ4e?format=jpg&name=orig', altText=None)], retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=None, cashtags=None, card=None, viewCount=2852, vibe=None), inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=['AdoptCat', 'AdoptAShelterCat', 'AdoptDontShop', 'AdoptDontBuy', 'cats', 'kittens', 'SaveThemAll', 'spay', 'neuter', 'TNR'], cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/belal4abty/status/1643024578652913666', date=datetime.datetime(2023, 4, 3, 22, 56, tzinfo=datetime.timezone.utc), rawContent='Again!! Ukrainian troops use Modified Drone Grenades destroy 16 Russian artillery 15 tank in Kherson\n#Canada #Germany #Bakhmut #Kyiv #Ukraine\nhttps://t.co/BKdjaD77X2', renderedContent='Again!! Ukrainian troops use Modified Drone Grenades destroy 16 Russian artillery 15 tank in Kherson\n#Canada #Germany #Bakhmut #Kyiv #Ukraine\ntwtubekb9.page.link/z8p6MG3U6mjF38…', id=1643024578652913666, user=User(username='belal4abty', id=1614799771889172480, displayname='بلال الرابطي', rawDescription='النيه الطيبة: فهي مفتاح باب الرزق.\nوالكلمه الطيبة: فهي مفتاح باب القلوب.', renderedDescription='النيه الطيبة: فهي مفتاح باب الرزق.\nوالكلمه الطيبة: فهي مفتاح باب القلوب.', descriptionLinks=None, verified=False, created=datetime.datetime(2023, 1, 16, 1, 40, 48, tzinfo=datetime.timezone.utc), followersCount=20, friendsCount=146, statusesCount=2830, favouritesCount=196, listedCount=1, mediaCount=12, location='ليبيا', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1614799920581541891/uYkBpClH_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/1614799771889172480/1673833555', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024578652913666, lang='en', source='<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App</a>', sourceUrl='https://mobile.twitter.com', sourceLabel='Twitter Web App', links=[TextLink(text='twtubekb9.page.link/z8p6MG3U6mjF38…', url='https://twtubekb9.page.link/z8p6MG3U6mjF389UA', tcourl='https://t.co/BKdjaD77X2', indices=(142, 165))], media=None, retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=['Canada', 'Germany', 'Bakhmut', 'Kyiv', 'Ukraine'], cashtags=None, card=SummaryCard(title='Again!! Ukrainian troops use Modified Drone Grenades destroy 16 Russian artillery 15 tank in Kherson', url='https://twtubekb9.page.link/z8p6MG3U6mjF389UA', description=None, thumbnailUrl='https://pbs.twimg.com/card_img/1641772000153829376/XThdDsY2?format=jpg&name=orig', siteUser=None, creatorUser=None), viewCount=None, vibe=None), Tweet(url='https://twitter.com/WildpferdZ/status/1643024567424675843', date=datetime.datetime(2023, 4, 3, 22, 55, 57, tzinfo=datetime.timezone.utc), rawContent='#ARD  „Können wir Krieg?“  \n\nIst das Euer Ernst, \nwollt Ihr nach dem Wegradieren\n der #Mutter \n\nnun unsere Söhne auch noch  in den #US #Stellvertreter #Ukraine Krieg drängen  ? \n#OERR im #Kriegstreiber\n #Propaganda Modus ? \n\nNicht von meinem  #Rundfunkbeitrag', renderedContent='#ARD  „Können wir Krieg?“  \n\nIst das Euer Ernst, \nwollt Ihr nach dem Wegradieren\n der #Mutter \n\nnun unsere Söhne auch noch  in den #US #Stellvertreter #Ukraine Krieg drängen  ? \n#OERR im #Kriegstreiber\n #Propaganda Modus ? \n\nNicht von meinem  #Rundfunkbeitrag', id=1643024567424675843, user=User(username='WildpferdZ', id=1517083951587811328, displayname='Soli darisch', rawDescription='#Makelovenotwar #ostdeutsch #gesund #NiemalsWaffen #FreeAssangeNow #TeamEmpathie #AufstandfuerFrieden #Meinungsfreiheit #initiativekindeswohl #Ampelmussweg', renderedDescription='#Makelovenotwar #ostdeutsch #gesund #NiemalsWaffen #FreeAssangeNow #TeamEmpathie #AufstandfuerFrieden #Meinungsfreiheit #initiativekindeswohl #Ampelmussweg', descriptionLinks=None, verified=False, created=datetime.datetime(2022, 4, 21, 10, 13, 9, tzinfo=datetime.timezone.utc), followersCount=1971, friendsCount=3719, statusesCount=15343, favouritesCount=30299, listedCount=8, mediaCount=1642, location='Winterfell ', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1627075010983768068/AsM3vOn-_normal.jpg', profileBannerUrl='https://pbs.twimg.com/profile_banners/1517083951587811328/1675810088', label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1643024567424675843, lang='de', source='<a href="https://mobile.twitter.com" rel="nofollow">Twitter Web App</a>', sourceUrl='https://mobile.twitter.com', sourceLabel='Twitter Web App', links=None, media=None, retweetedTweet=None, quotedTweet=None, inReplyToTweetId=None, inReplyToUser=None, mentionedUsers=None, coordinates=None, place=None, hashtags=['ARD', 'Mutter', 'US', 'Stellvertreter', 'Ukraine', 'OERR', 'Kriegstreiber', 'Propaganda', 'Rundfunkbeitrag'], cashtags=None, card=None, viewCount=None, vibe=None), Tweet(url='https://twitter.com/CharlieSnider12/status/1643024561300987904', date=datetime.datetime(2023, 4, 3, 22, 55, 55, tzinfo=datetime.timezone.utc), rawContent='@CatieGus @krassenstein He only follows yellow-Ukraine yellow-', renderedContent='@CatieGus @krassenstein He only follows yellow-Ukraine yellow-', id=1643024561300987904, user=User(username='CharlieSnider12', id=1598862058455535616, displayname='the iconoclast', rawDescription='Banned for life 4 times under the Twitter Tyranny Tyrants!! Welcome Mr “freedom” Musk', renderedDescription='Banned for life 4 times under the Twitter Tyranny Tyrants!! Welcome Mr “freedom” Musk', descriptionLinks=None, verified=False, created=datetime.datetime(2022, 12, 3, 2, 10, 15, tzinfo=datetime.timezone.utc), followersCount=2265, friendsCount=4074, statusesCount=3090, favouritesCount=806, listedCount=4, mediaCount=373, location='', protected=False, link=None, profileImageUrl='https://pbs.twimg.com/profile_images/1599589317931929601/48tujDdd_normal.jpg', profileBannerUrl=None, label=None), replyCount=0, retweetCount=0, likeCount=0, quoteCount=0, conversationId=1642973989231382528, lang='en', source='<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', sourceUrl='http://twitter.com/download/iphone', sourceLabel='Twitter for iPhone', links=None, media=None, retweetedTweet=None, quotedTweet=None, inReplyToTweetId=1642991103572942854, inReplyToUser=User(username='CatieGus', id=887094314, displayname='Hey Catie Text Trump to 88022', rawDescription=None, renderedDescription=None, descriptionLinks=None, verified=None, created=None, followersCount=None, friendsCount=None, statusesCount=None, favouritesCount=None, listedCount=None, mediaCount=None, location=None, protected=None, link=None, profileImageUrl=None, profileBannerUrl=None, label=None), mentionedUsers=[User(username='CatieGus', id=887094314, displayname='Hey Catie Text Trump to 88022', rawDescription=None, renderedDescription=None, descriptionLinks=None, verified=None, created=None, followersCount=None, friendsCount=None, statusesCount=None, favouritesCount=None, listedCount=None, mediaCount=None, location=None, protected=None, link=None, profileImageUrl=None, profileBannerUrl=None, label=None), User(username='krassenstein', id=133938408, displayname='Brian Krassenstein', rawDescription=None, renderedDescription=None, descriptionLinks=None, verified=None, created=None, followersCount=None, friendsCount=None, statusesCount=None, favouritesCount=None, listedCount=None, mediaCount=None, location=None, protected=None, link=None, profileImageUrl=None, profileBannerUrl=None, label=None)], coordinates=None, place=None, hashtags=None, cashtags=None, card=None, viewCount=None, vibe=None)]

Operating system

macos 13.2.1

Python version: output of python3 --version

Python 3.11.0

snscrape version: output of snscrape --version

snscrape 0.6.2.20230320

Scraper

TwitterSearchScraper

How are you using snscrape?

import snscrape.modules.twitter as sntwitter

Log output

adding the full output report (proxy address changed for privacy reasons):

INFO:snscrape.modules.twitter:Retrieving scroll page None
INFO:snscrape.modules.twitter:Retrieving guest token
INFO:snscrape.base:Retrieving https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): twitter.com:443
DEBUG:snscrape.base:Connected to: ('212.102.54.9', 20503)
DEBUG:snscrape.base:Connection cipher: ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)
DEBUG:urllib3.connectionpool:https://twitter.com:443 "GET /search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click HTTP/1.1" 200 None
INFO:snscrape.base:Retrieved https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click: 200
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:29 GMT', 'perf': '7626143928', 'expiry': 'Tue, 31 Mar 1981 05:00:00 GMT', 'pragma': 'no-cache', 'server': 'tsa_o', 'set-cookie': 'gt=1643056157370597377; Max-Age=10800; Expires=Tue, 04 Apr 2023 04:01:29 GMT; Path=/; Domain=.twitter.com; Secure, guest_id=v1%3A168057008906016788; Max-Age=34214400; Expires=Sat, 04 May 2024 01:01:29 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None, ct0=; Max-Age=-1680570088; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=Lax', 'content-type': 'text/html; charset=utf-8', 'x-powered-by': 'Express', 'cache-control': 'no-cache, no-store, must-revalidate, pre-check=0, post-check=0', 'last-modified': 'Tue, 04 Apr 2023 01:01:29 GMT', 'x-frame-options': 'DENY', 'x-transaction-id': '1830e7184ed219c3', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'content-security-policy': "connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://client-api.arkoselabs.com/ https://iframe.arkoselabs.com/  https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://maps.googleapis.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://client-api.arkoselabs.com/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js  'nonce-MzMzNGNmYjktMTcwYi00NTU1LWEzNjUtZWE1NTI2MTAxMjQw'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false", 'strict-transport-security': 'max-age=631138519', 'cross-origin-opener-policy': 'same-origin-allow-popups', 'cross-origin-embedder-policy': 'unsafe-none', 'content-encoding': 'gzip', 'x-response-time': '134', 'x-connection-hash': '6d80cc5af49cddcfb95ed53dc8077440350cafc4810cf8e7f65d9f07f12628e1', 'transfer-encoding': 'chunked'}
DEBUG:snscrape.base:https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click retrieved successfully
DEBUG:snscrape.modules.twitter:Found guest token in cookies
DEBUG:snscrape.modules.twitter:Using guest token 1643056157370597377
INFO:snscrape.base:Retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10', 'Authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA', 'Referer': 'https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click', 'Accept-Language': 'en-US,en;q=0.5', 'x-guest-token': '1643056157370597377'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com:443
DEBUG:snscrape.base:Connected to: ('46.105.116.15', 20503)
DEBUG:snscrape.base:Connection cipher: ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)
DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "GET /2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe HTTP/1.1" 404 92
INFO:snscrape.base:Retrieved https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: 404
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:30 UTC', 'perf': '7626143928', 'server': 'tsa_o', 'content-type': 'application/json;charset=utf-8', 'cache-control': 'no-cache, no-store, max-age=0', 'x-transaction-id': 'df30a41661616c41', 'x-rate-limit-limit': '250', 'x-rate-limit-reset': '1680570990', 'x-rate-limit-remaining': '249', 'strict-transport-security': 'max-age=631138519', 'content-encoding': 'gzip', 'content-length': '92', 'x-response-time': '103', 'x-connection-hash': '3e1c1eec8c1bfe04d508fe10c10e08adbcf4f2cd0312c7eddfb2c4031cd7cfae'}
INFO:snscrape.modules.twitter:Retrieving guest token
INFO:snscrape.base:Retrieving https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://twitter.com:443 "GET /search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click HTTP/1.1" 200 None
INFO:snscrape.base:Retrieved https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click: 200
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:30 GMT', 'perf': '7626143928', 'expiry': 'Tue, 31 Mar 1981 05:00:00 GMT', 'pragma': 'no-cache', 'server': 'tsa_o', 'set-cookie': 'gt=1643056165205540866; Max-Age=10800; Expires=Tue, 04 Apr 2023 04:01:30 GMT; Path=/; Domain=.twitter.com; Secure, ct0=; Max-Age=-1680570089; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=Lax', 'content-type': 'text/html; charset=utf-8', 'x-powered-by': 'Express', 'cache-control': 'no-cache, no-store, must-revalidate, pre-check=0, post-check=0', 'last-modified': 'Tue, 04 Apr 2023 01:01:30 GMT', 'x-frame-options': 'DENY', 'x-transaction-id': '42d3c906a53bbf2e', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'content-security-policy': "connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://client-api.arkoselabs.com/ https://iframe.arkoselabs.com/  https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://maps.googleapis.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://client-api.arkoselabs.com/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js  'nonce-MDU0OTEwY2MtN2U1MS00MTY4LWI1YzctMjVhNThkOTllZTM2'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false", 'strict-transport-security': 'max-age=631138519', 'cross-origin-opener-policy': 'same-origin-allow-popups', 'cross-origin-embedder-policy': 'unsafe-none', 'content-encoding': 'gzip', 'x-response-time': '131', 'x-connection-hash': '6d80cc5af49cddcfb95ed53dc8077440350cafc4810cf8e7f65d9f07f12628e1', 'transfer-encoding': 'chunked'}
DEBUG:snscrape.base:https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click retrieved successfully
DEBUG:snscrape.modules.twitter:Found guest token in cookies
DEBUG:snscrape.modules.twitter:Using guest token 1643056165205540866
INFO:snscrape.base:Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (404), retrying
INFO:snscrape.base:Waiting 1 seconds
INFO:snscrape.base:Retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10', 'Authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA', 'Referer': 'https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click', 'Accept-Language': 'en-US,en;q=0.5', 'x-guest-token': '1643056165205540866'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "GET /2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe HTTP/1.1" 404 92
INFO:snscrape.base:Retrieved https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: 404
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:32 UTC', 'perf': '7626143928', 'server': 'tsa_o', 'content-type': 'application/json;charset=utf-8', 'cache-control': 'no-cache, no-store, max-age=0', 'x-transaction-id': 'be6809c449bbb80d', 'x-rate-limit-limit': '250', 'x-rate-limit-reset': '1680570992', 'x-rate-limit-remaining': '249', 'strict-transport-security': 'max-age=631138519', 'content-encoding': 'gzip', 'content-length': '92', 'x-response-time': '103', 'x-connection-hash': '3e1c1eec8c1bfe04d508fe10c10e08adbcf4f2cd0312c7eddfb2c4031cd7cfae'}
INFO:snscrape.modules.twitter:Retrieving guest token
INFO:snscrape.base:Retrieving https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://twitter.com:443 "GET /search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click HTTP/1.1" 200 None
INFO:snscrape.base:Retrieved https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click: 200
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:33 GMT', 'perf': '7626143928', 'expiry': 'Tue, 31 Mar 1981 05:00:00 GMT', 'pragma': 'no-cache', 'server': 'tsa_o', 'set-cookie': 'gt=1643056174403575811; Max-Age=10800; Expires=Tue, 04 Apr 2023 04:01:33 GMT; Path=/; Domain=.twitter.com; Secure, ct0=; Max-Age=-1680570092; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=Lax', 'content-type': 'text/html; charset=utf-8', 'x-powered-by': 'Express', 'cache-control': 'no-cache, no-store, must-revalidate, pre-check=0, post-check=0', 'last-modified': 'Tue, 04 Apr 2023 01:01:33 GMT', 'x-frame-options': 'DENY', 'x-transaction-id': 'd830a507fa42f4a3', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'content-security-policy': "connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://client-api.arkoselabs.com/ https://iframe.arkoselabs.com/  https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://maps.googleapis.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://client-api.arkoselabs.com/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js  'nonce-MTM4Yjg4YzQtZDdmMy00MjdkLWI1ODMtNTdjOTQzNmFmMDdl'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false", 'strict-transport-security': 'max-age=631138519', 'cross-origin-opener-policy': 'same-origin-allow-popups', 'cross-origin-embedder-policy': 'unsafe-none', 'content-encoding': 'gzip', 'x-response-time': '134', 'x-connection-hash': '6d80cc5af49cddcfb95ed53dc8077440350cafc4810cf8e7f65d9f07f12628e1', 'transfer-encoding': 'chunked'}
DEBUG:snscrape.base:https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click retrieved successfully
DEBUG:snscrape.modules.twitter:Found guest token in cookies
DEBUG:snscrape.modules.twitter:Using guest token 1643056174403575811
INFO:snscrape.base:Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (404), retrying
INFO:snscrape.base:Waiting 2 seconds
INFO:snscrape.base:Retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10', 'Authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA', 'Referer': 'https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click', 'Accept-Language': 'en-US,en;q=0.5', 'x-guest-token': '1643056174403575811'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "GET /2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe HTTP/1.1" 404 92
INFO:snscrape.base:Retrieved https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: 404
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:35 UTC', 'perf': '7626143928', 'server': 'tsa_o', 'content-type': 'application/json;charset=utf-8', 'cache-control': 'no-cache, no-store, max-age=0', 'x-transaction-id': 'fa825eb9c02b43d0', 'x-rate-limit-limit': '250', 'x-rate-limit-reset': '1680570995', 'x-rate-limit-remaining': '249', 'strict-transport-security': 'max-age=631138519', 'content-encoding': 'gzip', 'content-length': '92', 'x-response-time': '103', 'x-connection-hash': '3e1c1eec8c1bfe04d508fe10c10e08adbcf4f2cd0312c7eddfb2c4031cd7cfae'}
INFO:snscrape.modules.twitter:Retrieving guest token
INFO:snscrape.base:Retrieving https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://twitter.com:443 "GET /search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click HTTP/1.1" 200 None
INFO:snscrape.base:Retrieved https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click: 200
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:36 GMT', 'perf': '7626143928', 'expiry': 'Tue, 31 Mar 1981 05:00:00 GMT', 'pragma': 'no-cache', 'server': 'tsa_o', 'set-cookie': 'gt=1643056187557007369; Max-Age=10800; Expires=Tue, 04 Apr 2023 04:01:36 GMT; Path=/; Domain=.twitter.com; Secure, ct0=; Max-Age=-1680570095; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=Lax', 'content-type': 'text/html; charset=utf-8', 'x-powered-by': 'Express', 'cache-control': 'no-cache, no-store, must-revalidate, pre-check=0, post-check=0', 'last-modified': 'Tue, 04 Apr 2023 01:01:36 GMT', 'x-frame-options': 'DENY', 'x-transaction-id': '37efed9c3673ab9a', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'content-security-policy': "connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://client-api.arkoselabs.com/ https://iframe.arkoselabs.com/  https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://maps.googleapis.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://client-api.arkoselabs.com/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js  'nonce-M2ExM2NhOGQtYTFhMi00Y2M4LTg1MWMtMGMyNDZjYzcxZDA2'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false", 'strict-transport-security': 'max-age=631138519', 'cross-origin-opener-policy': 'same-origin-allow-popups', 'cross-origin-embedder-policy': 'unsafe-none', 'content-encoding': 'gzip', 'x-response-time': '129', 'x-connection-hash': '6d80cc5af49cddcfb95ed53dc8077440350cafc4810cf8e7f65d9f07f12628e1', 'transfer-encoding': 'chunked'}
DEBUG:snscrape.base:https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click retrieved successfully
DEBUG:snscrape.modules.twitter:Found guest token in cookies
DEBUG:snscrape.modules.twitter:Using guest token 1643056187557007369
INFO:snscrape.base:Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (404), retrying
INFO:snscrape.base:Waiting 4 seconds
INFO:snscrape.base:Retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10', 'Authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs=1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA', 'Referer': 'https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click', 'Accept-Language': 'en-US,en;q=0.5', 'x-guest-token': '1643056187557007369'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://api.twitter.com:443 "GET /2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe HTTP/1.1" 404 92
INFO:snscrape.base:Retrieved https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: 404
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:41 UTC', 'perf': '7626143928', 'server': 'tsa_o', 'content-type': 'application/json;charset=utf-8', 'cache-control': 'no-cache, no-store, max-age=0', 'x-transaction-id': '07070d4a664231d0', 'x-rate-limit-limit': '250', 'x-rate-limit-reset': '1680571001', 'x-rate-limit-remaining': '249', 'strict-transport-security': 'max-age=631138519', 'content-encoding': 'gzip', 'content-length': '92', 'x-response-time': '104', 'x-connection-hash': '3e1c1eec8c1bfe04d508fe10c10e08adbcf4f2cd0312c7eddfb2c4031cd7cfae'}
INFO:snscrape.modules.twitter:Retrieving guest token
INFO:snscrape.base:Retrieving https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click
DEBUG:snscrape.base:... with headers: {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.5563.5279 Safari/537.10'}
DEBUG:snscrape.base:... with environmentSettings: {'proxies': OrderedDict([('https', 'http://10.10.10.10:10100')]), 'stream': False, 'verify': True, 'cert': None}
DEBUG:urllib3.connectionpool:https://twitter.com:443 "GET /search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click HTTP/1.1" 200 None
INFO:snscrape.base:Retrieved https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click: 200
DEBUG:snscrape.base:... with response headers: {'date': 'Tue, 04 Apr 2023 01:01:41 GMT', 'perf': '7626143928', 'expiry': 'Tue, 31 Mar 1981 05:00:00 GMT', 'pragma': 'no-cache', 'server': 'tsa_o', 'set-cookie': 'gt=1643056209396654080; Max-Age=10800; Expires=Tue, 04 Apr 2023 04:01:41 GMT; Path=/; Domain=.twitter.com; Secure, ct0=; Max-Age=-1680570100; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=Lax', 'content-type': 'text/html; charset=utf-8', 'x-powered-by': 'Express', 'cache-control': 'no-cache, no-store, must-revalidate, pre-check=0, post-check=0', 'last-modified': 'Tue, 04 Apr 2023 01:01:41 GMT', 'x-frame-options': 'DENY', 'x-transaction-id': 'add1baba2ecdcef8', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'content-security-policy': "connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://client-api.arkoselabs.com/ https://iframe.arkoselabs.com/  https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://maps.googleapis.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://client-api.arkoselabs.com/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js  'nonce-MjMzNjRiZjUtZDhmNi00NGUwLTg1NDAtOTA0OWIyZDY2NTYx'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false", 'strict-transport-security': 'max-age=631138519', 'cross-origin-opener-policy': 'same-origin-allow-popups', 'cross-origin-embedder-policy': 'unsafe-none', 'content-encoding': 'gzip', 'x-response-time': '135', 'x-connection-hash': '6d80cc5af49cddcfb95ed53dc8077440350cafc4810cf8e7f65d9f07f12628e1', 'transfer-encoding': 'chunked'}
DEBUG:snscrape.base:https://twitter.com/search?f=live&lang=en&q=ukraine&src=spelling_expansion_revert_click retrieved successfully
DEBUG:snscrape.modules.twitter:Found guest token in cookies
DEBUG:snscrape.modules.twitter:Using guest token 1643056209396654080
ERROR:snscrape.base:Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (404)
CRITICAL:snscrape.base:4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=ukraine&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.
CRITICAL:snscrape.base:Errors: blocked (404), blocked (404), blocked (404), blocked (404)

DrSocket avatar Apr 03 '23 22:04 DrSocket

Used a private browser window to connect to twitter.com and this worked normally.

That's not relevant. Does the search work? That is, do you get results on https://twitter.com/search?q=ukraine&f=live? The search has different restrictions than the site in general.

3.11

Not a possible output of python3 --version

snscrape 0.6.1.20230314

Not the current snscrape version.

adding the full output report:

This is not a complete debug log. It's just the exception traceback formatted formatted in a strange way. See README for instructions on obtaining a complete log.

JustAnotherArchivist avatar Apr 03 '23 23:04 JustAnotherArchivist

update: Snscrape updated to latest version, python version output in correct format. debug log updated according to README instructions tested: https://twitter.com/search?q=ukraine&f=live? using proxy in private browser, not logged to any twitter account works as expected, results appearing as without proxy.

DrSocket avatar Apr 04 '23 00:04 DrSocket

That log was not generated with the latest snscrape version.

But I just found a proxy where the search works with curl but not with latest dev snscrape, so yeah, there is indeed some kind of issue here.

JustAnotherArchivist avatar Apr 04 '23 00:04 JustAnotherArchivist

I just ran the log again, using snscrape 0.6.2.20230320. with import logging; logging.basicConfig(level = logging.DEBUG) updated the original bug report

DrSocket avatar Apr 04 '23 01:04 DrSocket

Do you have control over the proxy server? If so, what's the proxy software and can you run snscrape from there directly for a test?

JustAnotherArchivist avatar Apr 04 '23 02:04 JustAnotherArchivist

I don't have control of the proxy server and it doesn't have many options, I pretty much just have the address, port, username, password.

Ran tests like in snscrape using https://google.com, https://api.ipify.org, https://httpbin.org/ip, all work.

req = session.prepare_request(requests.Request('GET', <url>))
envset = session.merge_environment_settings(req.url, proxies, None, None, None)
resp = session.send(req, **envset).content.decode('utf-8')

also tried running a tor server locally, this is more easily reproducible. with tor running you can add: {'https': 'socks5h://localhost:9050'} as proxies.

DrSocket avatar Apr 04 '23 03:04 DrSocket

Yeah, I expected as much. I'll see if I can figure out a way to reproduce it with a simple self-hosted proxy server. Else debugging is going to be tricky. Tor is useless for this purpose because of its inherent variability of circuits etc. It also doesn't give any insight into the relevant details anyway.

JustAnotherArchivist avatar Apr 04 '23 03:04 JustAnotherArchivist

Do you have recommendations on self-hosted proxy for debugging? Trying mitmproxy now.

DrSocket avatar Apr 04 '23 15:04 DrSocket

I don't have much experience with the type of proxy at play here, so no. mitmproxy is nice for other things, but I'm not sure it applies here. It establishes its own TLS connection rather than just passing through the client's (snscrape's) connection attempt, which could distort the results.

JustAnotherArchivist avatar Apr 04 '23 15:04 JustAnotherArchivist

I'm getting the same error Errors: blocked (404), blocked (404), blocked (404), blocked (404) while running snscrape with a proxy, example code:

from snscrape.modules.twitter import TwitterUserScraper

proxies = {
    "http": "http://192.168.1.5:20102",
    "https": "http://192.168.1.5:20102",
}

u = TwitterUserScraper("ubuntu", proxies=proxies)
for i in u.get_items():
    print(i, type(i))

The proxy works fine and can be confirmed by:

import requests

proxies = {
    "http": "http://192.168.1.5:20102",
    "https": "http://192.168.1.5:20102",
}

resp = requests.get("https://twitter.com", proxies=proxies)
print(resp.status_code)
# 200

davuses avatar Apr 07 '23 12:04 davuses

The proxy works fine and can be confirmed by:

Loading the Twitter homepage isn't the same as loading the HTTP endpoints, I believe. If you can load whatever snscrape can't in a web browser (with no cookies - try an incognito or private window), that's different.

TheTechRobo avatar Apr 07 '23 14:04 TheTechRobo

Loading the Twitter homepage isn't the same as loading the HTTP endpoints, I believe. If you can load whatever snscrape can't in a web browser (with no cookies - try an incognito or private window), that's different.

I logged into the proxy server and ran the same code:

from snscrape.modules.twitter import TwitterUserScraper

u = TwitterUserScraper("ubuntu")
for i in u.get_items():
    print(i, type(i))

The error didn't occur, so I think this shouldn't be a problem with the IP.

davuses avatar Apr 07 '23 14:04 davuses

@davuses Thank you, that's helpful. What proxy server software are you using?

JustAnotherArchivist avatar Apr 08 '23 01:04 JustAnotherArchivist

@davuses Thank you, that's helpful. What proxy server software are you using?

I set up the proxy server with this tool: https://github.com/v2fly/v2ray-core

davuses avatar Apr 08 '23 13:04 davuses

Can confirm it is not an IP issue, used the same proxy for testing both TWINT and snscrape, snscrape gives the 404 errors, TWINT does not.

Owningle avatar Apr 10 '23 17:04 Owningle

Sorry for posting this here, but how do you pass the Proxy-Authorization header to the command line.

denisj avatar Apr 19 '23 14:04 denisj

I had the same problem using the TwitterProfileScraper, I made the following changes to the code in base.py file and twitter.py. Here it is in case it helps anyone.

I tested usage with and without a proxy on the TwitterProfileScraper in WireShark and can confirm requests are going through the proxy.

  1. Update the base class

    • Find the base.py file.

    • Update the __init__ constructor to take a proxy argument and assign self._proxy this argument.

      def __init__(self, *, retries=3, proxies=None, proxy=None):
      self._retries = retries
      self._proxies = proxies
      self._proxy = proxy
      self._session = requests.Session()
      self._session.mount("https://", _HTTPSAdapter())

    • Update the _request function to set the environment proxy variable to the proxy you provide.

      environmentSettings = self._session.merge_environment_settings(req.url, proxies, None, None, None)
# ADD THE NEXT IF CONDITION
if self._proxy:
  environmentSettings["proxies"] = {"http": self._proxy, "https": self._proxy}

    1. Update the TwitterAPIScraper class

      • Add a proxy as an optional argument

      • Pass the proxy to the super()__init__ call

        class _TwitterAPIScraper(snscrape.base.Scraper):
    def __init__(self, baseUrl, *, guestTokenManager=None, maxEmptyPages=0, proxy=None, **kwargs):
        super().__init__(retries=3, proxies=None, proxy=proxy, **kwargs)
        self._baseUrl = baseUrl
        if guestTokenManager is None:
            global _globalGuestTokenManager
            if _globalGuestTokenManager is None:
                _globalGuestTokenManager = GuestTokenManager()
            guestTokenManager = _globalGuestTokenManager
        self._guestTokenManager = guestTokenManager
        self._maxEmptyPages = maxEmptyPages
        self._apiHeaders = {
            "User-Agent": None,
            "Authorization": _API_AUTHORIZATION_HEADER,
            "Referer": self._baseUrl,
            "Accept-Language": "en-US,en;q=0.5",
        }
        adapter = _TwitterTLSAdapter()
        self._session.mount("https://twitter.com", adapter)
        self._session.mount("https://api.twitter.com", adapter)
        self._set_random_user_agent()

      1. Update the TwitterSearchScraper

        • Add proxy as an optional argument in the __init__ method.

        • Pass the proxy to the super().__init__() call.

          class TwitterSearchScraper(_TwitterAPIScraper):
    name = "twitter-search"

    def __init__(self, query, *, cursor=None, top=False, maxEmptyPages=20, proxy=None, **kwargs):
        if not query.strip():
            raise ValueError("empty query")
        # kwargs["maxEmptyPages"] = maxEmptyPages
        super().__init__(
            baseUrl="https://twitter.com/search?"
            + urllib.parse.urlencode({"f": "live", "lang": "en", "q": query, "src": "spelling_expansion_revert_click"}),
            maxEmptyPages=maxEmptyPages,
            proxy=proxy,
            **kwargs,
        )
        self._query = query  # Note: may get replaced by subclasses when using user ID resolution
        self._cursor = cursor
        self._top = top

      2. Update the TwitterUserScraper

        • Add proxy as an optional argument in the __init__ method.

        • Pass the proxy to the super().__init__() call.

          class TwitterUserScraper(TwitterSearchScraper):
    name = "twitter-user"

    def __init__(self, user, *, proxy=None, **kwargs):
        self._isUserId = isinstance(user, int)
        if not self._isUserId and not self.is_valid_username(user):
            raise ValueError("Invalid username")
        super().__init__(f"from:{user}", proxy=proxy, **kwargs)
        self._user = user
        self._baseUrl = (
            f"https://twitter.com/{self._user}" if not self._isUserId else f"https://twitter.com/i/user/{self._user}"
        )

nadSTC avatar May 01 '23 05:05 nadSTC

@nadSTC None of those changes should be necessary. The proxies kwarg is already available on all scrapers and ends up in the environmentSettings. You just added a second, more complicated way of specifying the proxy.

JustAnotherArchivist avatar May 01 '23 06:05 JustAnotherArchivist