botocore Timestamp out of range error for certificate that expires in 2049 (beyond the 2038 epoch time overflow)

Describe the bug I'm getting a ValueError: timestamp out of range for platform time_t error when trying to delete an IoT Thing certificate with an expiration date in 2049.

Since the timestamp is 2524607999 and EPOCH time overflows at 2147483647 (the year 2038). The timestamp conversion function datetime.datetime.fromtimestamp is throwing an error.

Steps to reproduce I'm using the SwetupAWS.py tool in Amazon-FreeRTOS.

Running the with the args delete_prereq for a thing with a certificate ending beyond 2038 will produce the error.

Expected behavior Expecting that the timestamp can parse properly, so that info about the cert can be returned and it can be deleted.

Debug logs

2021-11-09 01:39:18,030 - urllib3.connectionpool - DEBUG - https://iot.us-east-1.amazonaws.com:443 "GET /certificates/<redacted certificate id> HTTP/1.1" 200 1939
2021-11-09 01:39:18,031 - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '81329fe8-df58-4fa1-beb9-24b4f471d289', 'Content-Length': '1939', 'x-amz-apigw-id': '<redacted>', 'X-Amzn-Trace-Id': 'Root=<redacted>', 'Connection': 'keep-alive', 'Date': 'Tue, 09 Nov 2021 06:39:18 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'}
2021-11-09 01:39:18,031 - botocore.parsers - DEBUG - Response body:
{"certificateDescription":{"caCertificateId":null,"certificateArn":"arn:aws:iot:us-east-1:<redacted>:cert/<redacted certificate id>","certificateId":"<redacted certificate id>","certificateMode":"DEFAULT","certificatePem":"-----BEGIN CERTIFICATE-----\n<certificate redacted>\n-----END CERTIFICATE-----\n","creationDate":1.636144795746E9,"customerVersion":1,"generationId":"e194471e-e7aa-426b-b5cb-48fee703e7c7","lastModifiedDate":1.636144795746E9,"ownedBy":"<redacted>","previousOwnedBy":null,"status":"ACTIVE","transferData":{"acceptDate":null,"rejectDate":null,"rejectReason":null,"transferDate":null,"transferMessage":null},"validity":{"notAfter":2.524607999E9,"notBefore":1.636144675E9}}}

Traceback (most recent call last):
  File "SetupAWS.py", line 204, in <module>
    cleanup()
  File "SetupAWS.py", line 156, in cleanup
    delete_prereq()
  File "SetupAWS.py", line 125, in delete_prereq
    cert_obj = certs.Certificate(cert_id)
  File "certs.py", line 13, in __init__
    result = self.client.describe_certificate(certificateId=self.id)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/client.py", line 692, in _make_api_call
    operation_model, request_dict, request_context)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/client.py", line 711, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/endpoint.py", line 135, in _send_request
    request, operation_model, context)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/endpoint.py", line 167, in _get_response
    request, operation_model)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/endpoint.py", line 218, in _do_get_response
    response_dict, operation_model.output_shape)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 245, in parse
    parsed = self._do_parse(response, shape)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 813, in _do_parse
    self._add_modeled_parse(response, shape, final_parsed)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 822, in _add_modeled_parse
    self._parse_payload(response, shape, member_shapes, final_parsed)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 863, in _parse_payload
    body_parsed = self._parse_shape(shape, original_parsed)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 312, in _parse_shape
    return handler(shape, node)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 612, in _handle_structure
    raw_value)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 312, in _parse_shape
    return handler(shape, node)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 612, in _handle_structure
    raw_value)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 312, in _parse_shape
    return handler(shape, node)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 612, in _handle_structure
    raw_value)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 312, in _parse_shape
    return handler(shape, node)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/parsers.py", line 629, in _handle_timestamp
    return self._timestamp_parser(value)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/utils.py", line 716, in parse_timestamp
    return _parse_timestamp_with_tzinfo(value, tzinfo)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/botocore/utils.py", line 687, in _parse_timestamp_with_tzinfo
    return datetime.datetime.fromtimestamp(value, tzinfo())
ValueError: timestamp out of range for platform time_t

I did my best to include the relevant portions without including any compromising information.

Nov 09 '21 07:11 Rich-Nelson

The epoch overflow issue seems like it can be addressed by a method described here. https://stackoverflow.com/questions/65481616/python-overflowerror-creating-date-from-timestamp-in-32bit-system#answer-65494704

After replacing this line return datetime.datetime.fromtimestamp(value, tzinfo()) in the _parse_timestamp_with_tzinfo function with this

return datetime.datetime.fromtimestamp(0) + datetime.timedelta(seconds=value) With that change, timestamps beyond epoch time parse without error.

I suggest a solution along these lines to support a broader range of dates. Note: This fix does not currently include the timezone info as that was throwing an error.

Nov 09 '21 15:11 Rich-Nelson

Hi @Rich-Nelson,

Sorry to hear about the issues. I'm not sure if we can go that route in terms of a fix, but I'll review with the team to get their thoughts.

Nov 11 '21 00:11 stobrien89

Hi @Rich-Nelson,

I thought this looked familiar— We had this come up earlier this year in this issue. This is somewhat outside of our control, but the team is open to adding some additional logic to work around this.

Nov 11 '21 23:11 stobrien89