chat-downloader
chat-downloader copied to clipboard
xenova
[BUG] Unhandled `JSONDecodeError` when YouTube API returns 502
Basic information
- Program version: 0.2.0
- Python version: 3.9.4
- Operating system: AlmaLinux 8.6 (Sky Tiger) (4.18.0-348.20.1.el8_5.x86_64)
Describe the bug
This happened when I was downloading the chat of a live stream. The YouTube API returned 502 and the response content was not JSON but HTML, then JSONDecodeError was raised from the json module.
Command used
- The command used (I wasn't prepared for debugging and this rarely happens so it was not verbose log, sorry):
chat_downloader --message_groups "messages superchat" -o "live_chat-Q16l-aJaX58.json" -- "Q16l-aJaX58"
- Output from the above command:
Exception Traceback
Traceback (most recent call last):
File "/cs/home/hl85/usr/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/cs/home/hl85/usr/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/cs/home/hl85/usr/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/cs/home/hl85/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/cs/home/hl85/usr/lib/python3.9/site-packages/chat_downloader/sites/youtube.py", line 1303, in _get_continuation_info
json_response = response.json()
File "/cs/home/hl85/usr/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] <!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 502 (Server Error)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>502.</b> <ins>That’s an error.</ins>
<p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>
: 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/cs/home/hl85/usr/bin/cdl", line 8, in <module>
sys.exit(main())
File "/cs/home/hl85/usr/lib/python3.9/site-packages/chat_downloader/cli.py", line 194, in main
run(**args.__dict__)
File "/cs/home/hl85/usr/lib/python3.9/site-packages/chat_downloader/chat_downloader.py", line 360, in run
for message in chat:
File "/cs/home/hl85/usr/lib/python3.9/site-packages/chat_downloader/sites/common.py", line 283, in __next__
item = next(self.chat)
File "/cs/home/hl85/usr/lib/python3.9/site-packages/chat_downloader/sites/youtube.py", line 1668, in _get_chat_messages
yt_info = self._get_continuation_info(
File "/cs/home/hl85/usr/lib/python3.9/site-packages/chat_downloader/sites/youtube.py", line 1320, in _get_continuation_info
self.retry(attempt_number, error=e, **program_params,
The output was cut-off here, this was all I had in the terminal.
Expected behavior
The exception was caught and the request was retried, just like how other network errors were handled.
My memory is fuzzy, but if this is an AttributeError, then this is a bug in the "requests" module due to a bugged exception hierarchy and the JSONDecodeError "doc" attribute not being present when trying to access it here; this bug is present in requests 2.27.1 . It should be fixed with requests commit https://github.com/psf/requests/commit/fa1b0a367abc8488542f7ce7c02a3614ad8aa09d , which is in requests version 2.28.0, released in June.
I may be wrong, because from my memory, this bug would cause chat_downloader to crash.
I believe you can check your version of requests with python -m pip show requests .
OK I didn't actually read the source code and assumed it was just an unhandled exception. I'll investigate it further but it's kinda hard to get a HTTP 502 status back again from YouTube unless they blew up their server again.