reddit_scraper_and_sentiment_analyzer
reddit_scraper_and_sentiment_analyzer copied to clipboard
On some dates I get the following error:JSONDecodeError: Expecting value
I get the error below on some dates. Why that is and how can I fix it? Thank you :/
File "C:\Users\apavl\reddit\download_data_from_reddit.py", line 95, in <module>
data = get_pushshift_data(key_word, after, before, sub_reddit)
File "C:\Users\apavl\reddit\download_data_from_reddit.py", line 15, in get_pushshift_data
data = json.loads(r.text)
File "C:\Users\apavl\anaconda3\envs\reddit\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\apavl\anaconda3\envs\reddit\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\apavl\anaconda3\envs\reddit\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting value
Hi! Not sure if this is still relevant for you right now, but I "fixed" this issue by catching the JSONDecodeError where get_pushshift_data() is called in the main - in both lines 93 and 106 - and just repeated the method call. So something like this:
try: data = get_pushshift_data(key_word, after, before, sub_reddit) except json.decoder.JSONDecodeError: print('JSON decode error, retrying API request...') data = get_pushshift_data(key_word, after, before, sub_reddit)
This is obviously not a fix, but the error occurred very inconsistently for me and wouldn't occur two times in a row on the same API request. At the very least, it has made the process more stable for me.