twitterscraper
twitterscraper copied to clipboard
taspinar
Got 0 tweets today. Scraping correctly until yesterday!
My network blocks proxys, so even with --disableproxy initially I got
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='free-proxy-list.net', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002191C342A60>: Failed to establish a new connection: [WinError 10060]
After setting up a VPN to bypass the proxy block, am now getting
requests.exceptions.ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
i think they shut down non-javascipt render at twitter search web page
@dantechguy I was using Twint, and it stopped functioning as well today. https://github.com/twintproject/twint/issues/863
Same problem Getting 0 tweets every time I run the query It was working fine till yesterday If anyone can suggest some alternatives It'd be great!
I just changed the code to solve the problem. Check the changes at https://github.com/taspinar/twitterscraper/pull/337
i think they shut down non-javascipt render at twitter search web page
Maybe because they launched their new API: https://blog.twitter.com/developer/en_us/topics/tools/2020/introducing_new_twitter_api.html
A legacy feed is still available by submitting the form in the noscript tag. The scraping will be different though.
A legacy feed is still available by submitting the form in the noscript tag. The scraping will be different though.
how is this done?
A legacy feed is still available by submitting the form in the noscript tag. The scraping will be different though.
how is this done?
Never mind my comment, I missed that it's already fixed in #337.
Fix #337 did not completely fix the user search (keyword search seems good). Seems 1st return is JSON, but additional returns may not be. I added a "try: except:" after "try:" in line 107. It first trys response.text...than excepts: json_resp...
Fix #337 did not completely fix the user search (keyword search seems good). Seems 1st return is JSON, but additional returns may not be. I added a "try: except:" after "try:" in line 107. It first trys response.text...than excepts: json_resp...
I'm not following your fix for the user search. Can you share your code snippet for those lines?
It did not fix the problem. I think the problem might be that the "INIT_URL_USER"/"RELOAD_URL_USER" have changed in query.py. The INIT_URL_USER seems to work OK, but says "has more items" is FALSE.
Yeah it appears that #337 did not fully fix the issue. Before I made the changes in #337 I would get 0 tweets every time, but now, it will run for a while and then eventually start returning 0 tweets again.
It did not fix the problem. I think the problem might be that the "INIT_URL_USER"/"RELOAD_URL_USER" have changed in query.py. The INIT_URL_USER seems to work OK, but says "has more items" is FALSE.
Did the changes in #337 fix the problem temporarily for you too? With it I started getting tweets returned, but after 5-10 minutes it starting returning 0 tweets again.
The #337 fix works for KW searches. #337 does not fix User searches. The INIT_URL_USER returns up to 20 (?) of a user tweets. The return is JSON, and processed OK. The initial returns seems to always say "has more items" = False, which is not true in most cases. My guess is that the URLs for User search need to be updated like they were for KW search in #337.
The #337 fix works for KW searches. #337 does not fix User searches. The INIT_URL_USER returns up to 20 (?) of a user tweets. The return is JSON, and processed OK. The initial returns seems to always say "has more items" = False, which is not true in most cases. My guess is that the URLs for User search need to be updated like they were for KW search in #337.
I don't think the #337 fix works fully for KW searches either. I am only doing KW searches and they work for a bit but after a while (it seems to be a different amount of time each time) 0 tweets is returned
@christiangfv, You can edit your py file /opt/anaconda3/lib/python3.8/site-packages/twitterscrapper and go into the query.py script and change the code and this will solve your issue. I troubleshooter for a few hours last night and this was the best way to do it.
#337 worked a few times, and then I get this error message and only a fraction of my limit.
Failed to parse JSON while requesting "https://twitter.com/i/search/timeline?f=tweets&vertical=default&include_available_features=1&include_entities=1&reset_error_state=false&src=typd&max_position=thGAVUV0VFVBaAwKqhpsTgySEWgoCogZGk3sohEjUAFQAlAFUAFQAVARUAFQAA&q=arson%20OR%20firebug%20since%3A2019-12-12%20until%3A2019-12-26&l=english" Traceback (most recent call last): File "/opt/anaconda3/lib/python3.8/site-packages/twitterscraper/query.py", line 109, in query_single_page html = json_resp['items_html'] or '' KeyError: 'items_html'
I also get errors for line 104 and 182 which doesn't resolve until a reinstall twitterscraper and the commit request.
Can anyone help? I'm a first time coder. Very, very new to this stuff.
I just changed the code to solve the problem. Check the changes at #337
@snmishra Thanks! I'm using Query Search with User Search, this work for a while, sometimes i get a lot of tweets and sometimes i get 0 tweets! Sometimes i repeat the searches with the same keywords, Accounts and dates, and a i get different tweets.
Same problem Getting 0 tweets every time I run the query It was working fine till yesterday If anyone can suggest some alternatives It'd be great!
hey can uh get just user info? because it's giving syntax error
As of August 21, 0 tweets is still an issue. Hopefully, this can be addressed soon.
Seriously, nobody fixed this in like 3 months? Just had to stop sending a header in the initial request, I'll try to get a pullrequest out...
@EklipZgit
All twitter scrapers dont seem to work anymore. I tried to use selenium to simply scrap maximum tweets between two chosen dates for given quiries. Check my work on : Scweet. let me know if you need any clarification.
the same problem got 0 tweets, maybe twitter launch their new API
have you solved this issue?
