50X error responses for post requests cause JSONDecodeError, breaking uploads
Here's the stack trace:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/usr/local/lib/python3.5/site-packages/pneumatic/uploader.py", line 137, in request
upload_response = json.loads(r.text)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "bulk_uploads.py", line 51, in <module>
data={'topic': 'stuff'})
File "/usr/local/lib/python3.5/site-packages/pneumatic/uploader.py", line 232, in upload
p.map(self.request, cleared_uploads)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I believe this line is the culprit, because if there isn't a proper JSON response from DocCloud API (as in a 200 status), then it can't convert the response (ie - HTML) to a dict.
Testing a fix now and hope to send a PR shortly.
So the platform is supposed to send a format-specific error response (JSON for JSON requests, JS for JS requests, or HTML for everything else). If it's sending HTML to an upload.json hit, then either nginx is throwing the 500 upstream, or Rails is fouling up the format negotiation. The former seems more likely. We should investigate and fix on DC why this was returning HTML, but yeah, you may want to protect against receiving non-JSON from the API just in case.