PyCap
PyCap copied to clipboard
redcap-tools
JSONDecodeError
Hello all,
I have been using this API for more than 6 months in our hospital, but it suddenly stopped working. I checked with some of my coworkers who were also using it.
We cannot even issue the basic command to export the records from our different RedCaps.
I have checked that the API key is correct, and made sure that there wasn't any significant change in our code that could affect the function of the script. We also didn't do any significant change to our development environment.
However, when we try to run:
rdf = project.export_records('df', record_type='flat', raw_or_label='label')
We get:
JSONDecodeError Traceback (most recent call last) Input In [30], in <cell line: 1>() ----> 1 rdf = project.export_records('df', record_type='flat', raw_or_label='label')
File ~\Anaconda3\lib\site-packages\redcap\methods\records.py:181, in Records.export_records(self, format_type, records, fields, forms, events, raw_or_label, raw_or_label_headers, event_name, record_type, export_survey_fields, export_data_access_groups, export_checkbox_labels, filter_logic, date_begin, date_end, decimal_character, export_blank_for_gray_form_status, df_kwargs) 176 # pylint: enable=line-too-long 177 payload: Dict[str, Any] = self._initialize_payload( 178 content="record", format_type=format_type, record_type=record_type 179 ) --> 181 fields = self._backfill_fields(fields, forms) 183 keys_to_add = ( 184 records, 185 fields, (...) 196 export_blank_for_gray_form_status, 197 ) 199 str_keys = ( 200 "records", 201 "fields", (...) 212 "exportBlankForGrayFormStatus", 213 )
File ~\Anaconda3\lib\site-packages\redcap\methods\records.py:47, in Records._backfill_fields(self, fields, forms) 44 return fields + [self.def_field] 46 if not fields: ---> 47 return self.field_names + [f"{ form }_complete" for form in self.forms] 49 return fields
File ~\Anaconda3\lib\site-packages\redcap\methods\base.py:100, in Base.field_names(self) 94 """Project field names 95 96 Note: 97 These are survey field names, not export field names 98 """ 99 if self._field_names is None: --> 100 self._field_names = self._filter_metadata(key="field_name") 102 return self._field_names
File ~\Anaconda3\lib\site-packages\redcap\methods\base.py:254, in Base._filter_metadata(self, key, field_name) 252 return "" 253 else: --> 254 res = [row[key] for row in self.metadata] 256 return res
File ~\Anaconda3\lib\site-packages\redcap\methods\base.py:80, in Base.metadata(self) 78 if self._metadata is None: 79 payload = self._initialize_payload("metadata", format_type="json") ---> 80 self._metadata = cast(Json, self._call_api(payload, return_type="json")) 82 return self._metadata
File ~\Anaconda3\lib\site-packages\redcap\methods\base.py:436, in Base._call_api(self, payload, return_type, file) 433 return_headers = return_type == "file_map" 435 rcr = _RCRequest(url=self.url, payload=payload, config=config) --> 436 return rcr.execute( 437 verify_ssl=self.verify_ssl, 438 return_headers=return_headers, 439 file=file, 440 **self._request_kwargs, 441 )
File ~\Anaconda3\lib\site-packages\redcap\request.py:189, in _RCRequest.execute(self, verify_ssl, return_headers, file, **kwargs) 165 """Execute the API request and return data 166 167 Args: (...) 183 exist, field doesn't exist, etc. 184 """ 185 response = self.session.post( 186 self.url, data=self.payload, verify=verify_ssl, files=file, **kwargs 187 ) --> 189 content = self.get_content( 190 response, 191 format_type=self.fmt, 192 return_empty_json=self.config.return_empty_json, 193 return_bytes=self.config.return_bytes, 194 ) 196 if self.fmt == "json": 197 try:
File ~\Anaconda3\lib\site-packages\redcap\request.py:153, in _RCRequest.get_content(response, format_type, return_empty_json, return_bytes) 150 return [{}] 152 if format_type == "json": --> 153 return response.json() 155 # don't do anything to csv/xml strings 156 return response.text
File ~\Anaconda3\lib\site-packages\requests\models.py:897, in Response.json(self, **kwargs) 891 except UnicodeDecodeError: 892 # Wrong UTF codec detected; usually because it's not UTF-8 893 # but some other 8-bit codec. This is an RFC violation, 894 # and the server didn't bother to tell us what codec was 895 # used. 896 pass --> 897 return complexjson.loads(self.text, **kwargs)
File ~\Anaconda3\lib\json_init_.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 341 s = s.decode(detect_encoding(s), 'surrogatepass') 343 if (cls is None and object_hook is None and 344 parse_int is None and parse_float is None and 345 parse_constant is None and object_pairs_hook is None and not kw): --> 346 return _default_decoder.decode(s) 347 if cls is None: 348 cls = JSONDecoder
File ~\Anaconda3\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
332 def decode(self, s, _w=WHITESPACE.match):
333 """Return the Python representation of s (a str instance
334 containing a JSON document).
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):
File ~\Anaconda3\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx) 353 obj, end = self.scan_once(s, idx) 354 except StopIteration as err: --> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Could you give us a clue on how to address this issue?
Thank you.
Best,
Edgar
Great to hear that you and your team have enjoyed using the package so far.
Which version of PyCap are you using? If it's not latest version (2.4.0), I suggest trying the latest version.
If it is the latest version, I suggest using a previous version and seeing if the error goes away
However -- you said you just started having issues. I haven't released to PyPI since March 28th 2023. So I suspect that this has to do with your site's REDCap version -- perhaps the newest/your version of REDCap breaks the assumptions PyCap makes about it's API.
What version of REDCap is your site using? And did they recently change it?
Hi @pwildenhain, I started to use PyCap to export records from our REDCap instance. I was running data = project.export_records() and it throw me the same JSONDecodeError described by @EdgarHSJD . My REDCap version is 13.5.1 and it's recently updated. Any suggestion is recommended.
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Thanks for reporting -- I'll look into this.
In the meantime you can use the format_type parameter in export_records to export to a different format to avoid the JSONDecodeError. Available options include 'df' (a pandas DataFrame), 'csv', or 'xml'.
Great to hear that you and your team have enjoyed using the package so far.
Which version of PyCap are you using? If it's not latest version (2.4.0), I suggest trying the latest version.
If it is the latest version, I suggest using a previous version and seeing if the error goes away
However -- you said you just started having issues. I haven't released to PyPI since March 28th 2023. So I suspect that this has to do with your site's REDCap version -- perhaps the newest/your version of REDCap breaks the assumptions PyCap makes about it's API.
What version of REDCap is your site using? And did they recently change it?
Hello @pwildenhain, I am using the last version of the API. Our RedCap Manager has confirmed that it was recently updated, which also caused the native API to stop working properly. The current RedCap version we are using is 13.4.11.
I haven't got any other update. I will follow your recommendation for exporting records, and will get back as soon as I get any update.
Thank you!!
@pwildenhain thanks for your instruction. I tried adding format_type parameter but it returns the same error. Here are the codes I ran:
import redcap
project = redcap.Project('url', 'token')
data = project.export_records(format_type='df')
The error message I got is
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/methods/records.py", line 181, in export_records
fields = self._backfill_fields(fields, forms)
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/methods/records.py", line 47, in _backfill_fields
return self.field_names + [f"{ form }_complete" for form in self.forms]
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/methods/base.py", line 100, in field_names
self._field_names = self._filter_metadata(key="field_name")
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/methods/base.py", line 254, in _filter_metadata
res = [row[key] for row in self.metadata]
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/methods/base.py", line 80, in metadata
self._metadata = cast(Json, self._call_api(payload, return_type="json"))
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/methods/base.py", line 436, in _call_api
return rcr.execute(
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/request.py", line 189, in execute
content = self.get_content(
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/redcap/request.py", line 153, in get_content
return response.json()
File "/Users/PycharmProjects/REDCapAPI/venv/lib/python3.8/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Ah right, that's because I hit the API to get the metadata in JSON format, so you'll still get this error further down the stack trace.
Unfortunately since this is a REDCap bug there's not much I can do about it. If the errors persist I recommend temporarily switch bing over to using the requests package to get data (you can find code examples in the API playground of your project)
Ah right, that's because I hit the API to get the metadata in JSON format, so you'll still get this error further down the stack trace.
Unfortunately since this is a REDCap bug there's not much I can do about it. If the errors persist I recommend temporarily switch bing over to using the
requestspackage to get data (you can find code examples in the API playground of your project)
Dear pwildenhain,
I have the same issue reported by @EdgarHSJD and @xj2193 and I tried also what you said in your last comment (I executed one of the API examples from RedCap) and still does not work.
Yea if the examples in the API playground aren't working, that means something is fundamentally broken with the API in the version of REDCap that you're using. I suggest asking your REDCap admins if they can roll back the version or install a later version if there is a bug fix release out already
Hello all,
Our RedCap admin recently updated it to version 13.6.0, and we still have the same problem. We cannot roll back to a previous version for security reasons.
Do you have any other suggestion on how to address this issue?
Thank you.
Best
Hello all,
Our RedCap admin recently updated it to version 13.6.0, and we still have the same problem. We cannot roll back to a previous version for security reasons.
Do you have any other suggestion on how to address this issue?
Thank you.
Best
I'm not sure what the issue could be. Right now I can test on 13.5.0 and on 13.5.4 without any issue.
@EdgarHSJD does the issue still persist if you try the python code example from the API Playground? If there is something that PyCap can do to help workaround this, then I'm happy to assist with that.
How is the status of this issue? I am getting the same error... raceback (most recent call last): File "/data/xnat-phrt/venv-xnat/lib64/python3.9/site-packages/requests/models.py", line 974, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python3.9/json/init.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/phrt/src/REDCap.py", line 17, in
Hi Maria,
Can you try running the sample Python code in the API playground to export records and seeing if that works or not?
And can you report what version of REDCap you're using? Because if it's the latest maybe I can test it out again since it's been a while
I tried the code in the API playground and does not work. I also tried to reach the API via curl to aassure the endpoint is correct and that seems to work.
And the REDCap version is v14.1.0 (I need to stick to this, the management does not depend of me)
I also tried to traceback. It seems the header from the REDCap API get response changed the format (maybe schema, or space in the response) and It is not able to decode the response as JSON compatible string
Thanks in advance fr your support!
The administrators agreed to upgrade to the last version of REDCap to v14.3.14 so I will test soon against and update status
I also realized the test suite hadn't ran in like 8 months, so I just re-ran it manually and everything passed (including tests related to exporting records) so I'm pretty convinced this is a redcap versioning issue