twitter-scraper JSONDecodeError: Expecting value: line 1 column 1 (char 0)

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Open RasmusFonseca opened this issue 3 years ago • 21 comments

Running the minimal example crashes with an error:

$ ipython
Python 3.8.3 (default, May 19 2020, 13:54:14) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from twitter_scraper import get_tweets                                                                                        

In [2]: for tweet in get_tweets('twitter', pages=1): 
   ...:     print(tweet['text']) 
   ...:                                                                                                                               
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-2-bd0d769d18f8> in <module>
----> 1 for tweet in get_tweets('twitter', pages=1):
      2     print(tweet['text'])
      3 

~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in get_tweets(query, pages)
    169             pages += -1
    170 
--> 171     yield from gen_tweets(pages)
    172 
    173 

~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in gen_tweets(pages)
     34         while pages > 0:
     35             try:
---> 36                 json_response = request.json()
     37                 html = HTML(
     38                     html=json_response["items_html"], url="bunk", default_encoding="utf-8"

~/opt/miniconda3/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
    896                     # used.
    897                     pass
--> 898         return complexjson.loads(self.text, **kwargs)
    899 
    900     @property

~/opt/miniconda3/lib/python3.8/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    355             parse_int is None and parse_float is None and
    356             parse_constant is None and object_pairs_hook is None and not kw):
--> 357         return _default_decoder.decode(s)
    358     if cls is None:
    359         cls = JSONDecoder

~/opt/miniconda3/lib/python3.8/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/opt/miniconda3/lib/python3.8/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Aug 23 '20 11:08 RasmusFonseca

Same here - python 3.7.4

Aug 25 '20 18:08 Avishay116

same issue. python 3.7.6

Aug 25 '20 20:08 gonhonoria

The response might be empty due to the request failed.

Aug 26 '20 07:08 yimingStar

ok, Why would the request failed? I have tried with a bunch of account names but still getting the same error. Has there been any change on the script?

Aug 26 '20 12:08 gonhonoria

While trying parsing from Hashtags, it works. The issue is specific to parsing profile tweets

Aug 26 '20 12:08 gonhonoria

It is because the url request for the user profile is somehow unavailable. See issue #166, the user-agent that this module is using cannot get the proper response now. Maybe Twitter has banned the user-agent, or maybe Twitter has changed all their website into Client Side Rendering causing request-html failed to get the elements that it usually could.

Aug 26 '20 14:08 yimingStar

@yimingStar Agree. Looks like twitter has made some major changes.

Sep 02 '20 14:09 botbotbotbotboot

If anyone interested in a workaround, please check this out @yimingStar @bisguzar

https://github.com/Foo-Manroot/tweet-feed/issues/1#issuecomment-686635709

Dont know if this method is good enough.

Sep 04 '20 06:09 botbotbotbotboot

Here is my attempt at a solution based on @Foo-Manroot instructions. https://gist.github.com/dgnsrekt/8cdb0c10e4a8fe6dcef65ee3a66a4e33 I'm having an issue with getting rate limited. Once, I can get past that issue I can make it nice and pretty and submit a pull request.

Sep 05 '20 01:09 dgnsrekt

Same issue

Sep 06 '20 23:09 magicaltoast

Found another source of tweets - widgets at https://publish.twitter.com/ Get JSON from https://syndication.twitter.com/timeline/profile?screen_name=<NAME> The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.

Sep 19 '20 08:09 n0madic

Found another source of tweets - widgets at https://publish.twitter.com/ Get JSON from https://syndication.twitter.com/timeline/profile?screen_name=<NAME> The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.

Could you tell me how to do that! Many thanks!

Sep 21 '20 09:09 Nicknameinmel

Everything is very simple - we take JSON from the link, take from it the “body” field with the timeline and parse it as usual. JSON also has "minPosition" and "maxPosition" headers for pagination. Implemented in my Go project https://github.com/n0madic/twitter-scraper.

Sep 21 '20 09:09 n0madic

I have the same issue on Python 3.8.2. Everyone know other library for scrapper or know If the new API twitter version allows to get tweets from more than seven days ago? Thanks a lot.