requests-arcgis-auth
requests-arcgis-auth copied to clipboard
Resolve a performance hit when token is expired and the download content is large
Problem Statement: When downloading a large payload, there is a significant delay due to the response hook. The code has a 'response hook' registered where it scans the response for an expired token. If the token is expired, it re-acquires it and re-issues the same request. When the payload is very large, the logic in the response hook adds a significant performance delay while it tries to scan the JSON content for an expired token.
EX - Exporting a Hosted Feature Service (HFS) to a File Geodatabase (FGDB) ends up creating the FGDB in the users content directory. Attempting to download the FGDB 'data' demonstrates this issue. On a sample test - When the response_hook was disabled the download consistently completed in 1.5-3 seconds while it was taking 15-18 seconds with the response hook enabled. This example had a response content-length of 1579064 Bytes (~1.5MB)
consider a better solution to resolve this.
Example fix in the arcgis_saml_auth.py file where it only checks the response content if its less than 1KB:
def _handle_response(self, resp, **kwargs):
# type(r) = Response
# Check the response for an expired token... re-acquire if necessary
# ex: {u'error': {u'code': 498, u'details': [], u'message': u'Invalid token.'}}
### Handling Expired Tokens!!!
# Check for actual HTTP Status code 498 (when f=json is not supplied)
if resp.status_code == 498:
return self._handle_expired_token(resp,**kwargs)
# Only check response if its small (less than 1KB (1024 bytes)). If the response is large, it most likely is not an expired token and will degrade performance
check_response=False
if "Content-Length" in resp.headers:
if int(resp.headers.get("Content-Length")) <= 1024:
check_response=True
# Check for JSON error (vendor spec)
if check_response:
try:
if resp.json().get("error") is not None:
err = resp.json().get("error")
if err.get("code") == 498 and err.get("message") == "Invalid token.":
return self._handle_expired_token(resp, **kwargs)
else:
# Why do we get here?!?!?!?
# {u'error': {u'code': 403,
# u'details': [],
# u'message': u'You do not have permissions to access this resource or perform this operation.',
# u'messageCode': u'GWM_0003'}}
# OTHERS?!?!?!?
# raise TokenAuthenticationError("Failed to handle expired token...")
pass
# Unable to parse JSON data... requestor could ask for non-JSON formatted data... Just throw away exception for now...
except ValueError:
pass
# Unable to parse JSON data due to a memory error. Requestor could ask for a large payload that is not JSON response. Throw away exception for now and assume the token was still valid.
except MemoryError:
pass
# Unable to parse JSON data due to a memory error. Requestor could ask for a large payload that is not JSON response. Throw away exception for now and assume the token was still valid.
except OverflowError:
pass
return resp