reddit_scraper_and_sentiment_analyzer icon indicating copy to clipboard operation
reddit_scraper_and_sentiment_analyzer copied to clipboard

On some dates I get the following error:JSONDecodeError: Expecting value

Open apavlo89 opened this issue 4 years ago • 1 comments

I get the error below on some dates. Why that is and how can I fix it? Thank you :/

  File "C:\Users\apavl\reddit\download_data_from_reddit.py", line 95, in <module>
    data = get_pushshift_data(key_word, after, before, sub_reddit)

  File "C:\Users\apavl\reddit\download_data_from_reddit.py", line 15, in get_pushshift_data
    data = json.loads(r.text)

  File "C:\Users\apavl\anaconda3\envs\reddit\lib\json\__init__.py", line 348, in loads
    return _default_decoder.decode(s)

  File "C:\Users\apavl\anaconda3\envs\reddit\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "C:\Users\apavl\anaconda3\envs\reddit\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value

apavlo89 avatar Dec 10 '20 12:12 apavlo89

Hi! Not sure if this is still relevant for you right now, but I "fixed" this issue by catching the JSONDecodeError where get_pushshift_data() is called in the main - in both lines 93 and 106 - and just repeated the method call. So something like this:

try: data = get_pushshift_data(key_word, after, before, sub_reddit) except json.decoder.JSONDecodeError: print('JSON decode error, retrying API request...') data = get_pushshift_data(key_word, after, before, sub_reddit)

This is obviously not a fix, but the error occurred very inconsistently for me and wouldn't occur two times in a row on the same API request. At the very least, it has made the process more stable for me.

sb-barbarossa avatar Jan 18 '21 23:01 sb-barbarossa