boto3 icon indicating copy to clipboard operation
boto3 copied to clipboard

Published 20 hours ago • verified publisher icon boto

Timestream client does not accept TIMESTAMP as MeasureValueType

Open BastianZim opened this issue 3 years ago • 8 comments

Describe the bug A clear and concise description of what the bug is.

Timestream accepts TIMESTAMP as MeasureValueType according to the docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/timestream-write.html#TimestreamWrite.Client.write_records

When writing a record with this value type, the write fails with

Traceback (most recent call last):

  File "/var/folders/j5/ym3yhfx510d4v_cnz9j6rmd80000gn/T/ipykernel_75705/577096492.py", line 1, in <module>
    _write_to_timestream(df)

  File "/Users/user/GitHub/folder/software/file.py", line 213, in _write_to_timestream
    result = client.write_records(DatabaseName="name",

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type.

Steps to reproduce If you have a runnable example, please include it as a snippet or link to a repository/gist for larger code examples.

Sample record:

[{'MeasureName': 'Test date', 'MeasureValueType': 'TIMESTAMP', 'MeasureValue': '1643987700000000000'}, {'MeasureName': 'Test value', 'MeasureValueType': 'DOUBLE', 'MeasureValue': '184532.6961294596385'}]

Sample Common Attributes:

{'Dimensions': [{'Name': 'name', 'Value': '0928.9274'}, {'Name': 'currency', 'Value': 'USD'}, {'Name': 'type', 'Value': 'cash'}], 'Time': '1643932800', 'TimeUnit': 'SECONDS'}

Sample call:

result = t_client.write_records(DatabaseName="db",
                                        TableName="table",
                                        Records=records_list,
                                        CommonAttributes=common_attributes)

(records_list is Sample record from above and common_attributes is Sample Common Attributes.)

Error:

Traceback (most recent call last):

  File "/var/folders/j5/ym3yhfx510d4v_cnz9j6rmd80000gn/T/ipykernel_75705/577096492.py", line 1, in <module>
    _write_to_timestream(df)

  File "/Users/user/GitHub/folder/software/file.py", line 213, in _write_to_timestream
    result = client.write_records(DatabaseName="name",

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/opt/homebrew/Caskroom/mambaforge/base/envs/name/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type.

Expected behavior A clear and concise description of what you expected to happen.

The record should be successfully written to Timestream.

Debug logs Full stack trace by adding boto3.set_stream_logger('') to your code.

2022-02-08 17:12:18,836 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.WriteRecords: calling handler <bound method EndpointDiscoveryHandler.gather_identifiers of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,837 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.WriteRecords: calling handler <function generate_idempotent_uuid at 0x10cb66dc0>
2022-02-08 17:12:18,838 botocore.hooks [DEBUG] Event before-call.timestream-write.WriteRecords: calling handler <function add_recursion_detection_header at 0x10cb66a60>
2022-02-08 17:12:18,838 botocore.hooks [DEBUG] Event before-call.timestream-write.WriteRecords: calling handler <function inject_api_version_header_if_needed at 0x10cb6d670>
2022-02-08 17:12:18,839 botocore.endpoint [DEBUG] Making request for OperationModel(name=WriteRecords) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Timestream_20181101.WriteRecords', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50'}, 'body': b'{"DatabaseName": "db", "TableName": "table", "Records": [{"MeasureName": "Test date", "MeasureValueType": "TIMESTAMP", "MeasureValue": "1643987700000000000"}, {"MeasureName": "Test value", "MeasureValueType": "DOUBLE", "MeasureValue": "184532.6961294596385"}], "CommonAttributes": {"Dimensions": [{"Name": "name", "Value": "0928.9274"}, {"Name": "currency", "Value": "USD"}, {"Name": "type", "Value": "cash"}], "Time": "1643932800", "TimeUnit": "SECONDS"}}', 'url': 'https://ingest.timestream.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x1ba455790>, 'has_streaming_input': False, 'auth_type': None, 'discovery': {'identifiers': {}}}}
2022-02-08 17:12:18,840 botocore.hooks [DEBUG] Event request-created.timestream-write.WriteRecords: calling handler <bound method EndpointDiscoveryHandler.discover_endpoint of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,840 botocore.discovery [DEBUG] Discovering endpoints with kwargs: {}
2022-02-08 17:12:18,841 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.DescribeEndpoints: calling handler <bound method EndpointDiscoveryHandler.gather_identifiers of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,841 botocore.hooks [DEBUG] Event before-parameter-build.timestream-write.DescribeEndpoints: calling handler <function generate_idempotent_uuid at 0x10cb66dc0>
2022-02-08 17:12:18,841 botocore.hooks [DEBUG] Event before-call.timestream-write.DescribeEndpoints: calling handler <function add_recursion_detection_header at 0x10cb66a60>
2022-02-08 17:12:18,842 botocore.hooks [DEBUG] Event before-call.timestream-write.DescribeEndpoints: calling handler <function inject_api_version_header_if_needed at 0x10cb6d670>
2022-02-08 17:12:18,842 botocore.endpoint [DEBUG] Making request for OperationModel(name=DescribeEndpoints) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'Timestream_20181101.DescribeEndpoints', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50', 'x-amz-api-version': '2018-11-01'}, 'body': b'{}', 'url': 'https://ingest.timestream.eu-central-1.amazonaws.com/', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x1ba455790>, 'has_streaming_input': False, 'auth_type': None}}
2022-02-08 17:12:18,843 botocore.hooks [DEBUG] Event request-created.timestream-write.DescribeEndpoints: calling handler <bound method EndpointDiscoveryHandler.discover_endpoint of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:18,843 botocore.hooks [DEBUG] Event request-created.timestream-write.DescribeEndpoints: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1ba455520>>
2022-02-08 17:12:18,844 botocore.hooks [DEBUG] Event choose-signer.timestream-write.DescribeEndpoints: calling handler <function set_operation_specific_signer at 0x10cb66ca0>
2022-02-08 17:12:18,845 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-02-08 17:12:18,845 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:ingest.timestream.eu-central-1.amazonaws.com
x-amz-api-version:2018-11-01
x-amz-date:20220208T161218Z
x-amz-target:Timestream_20181101.DescribeEndpoints

content-type;host;x-amz-api-version;x-amz-date;x-amz-target

2022-02-08 17:12:18,846 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20220208T161218Z
20220208/eu-central-1/timestream/aws4_request
2022-02-08 17:12:18,846 botocore.auth [DEBUG] Signature:
2022-02-08 17:12:18,847 botocore.hooks [DEBUG] Event request-created.timestream-write.DescribeEndpoints: calling handler <function add_retry_headers at 0x10cb6dd30>
2022-02-08 17:12:18,847 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ingest.timestream.eu-central-1.amazonaws.com/, headers={'X-Amz-Target': b'Timestream_20181101.DescribeEndpoints', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50', 'x-amz-api-version': b'2018-11-01', 'X-Amz-Date': b'20220208T161218Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=//eu-central-1/timestream/aws4_request, SignedHeaders=content-type;host;x-amz-api-version;x-amz-date;x-amz-target, Signature=', 'amz-sdk-invocation-id': b'', 'amz-sdk-request': b'attempt=1', 'Content-Length': '2'}>
2022-02-08 17:12:18,848 botocore.httpsession [DEBUG] Certificate path: /usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/certifi/cacert.pem
2022-02-08 17:12:18,849 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): ingest.timestream.eu-central-1.amazonaws.com:443
2022-02-08 17:12:19,031 urllib3.connectionpool [DEBUG] https://ingest.timestream.eu-central-1.amazonaws.com:443 "POST / HTTP/1.1" 200 108
2022-02-08 17:12:19,033 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': '', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '108', 'Date': 'Tue, 08 Feb 2022 16:12:18 GMT'}
2022-02-08 17:12:19,033 botocore.parsers [DEBUG] Response body:
b'{"Endpoints":[{"Address":"ingest-cell1.timestream.eu-central-1.amazonaws.com","CachePeriodInMinutes":1440}]}'
2022-02-08 17:12:19,034 botocore.hooks [DEBUG] Event needs-retry.timestream-write.DescribeEndpoints: calling handler <botocore.retryhandler.RetryHandler object at 0x1ba46f160>
2022-02-08 17:12:19,034 botocore.retryhandler [DEBUG] No retry needed.
2022-02-08 17:12:19,035 botocore.hooks [DEBUG] Event needs-retry.timestream-write.DescribeEndpoints: calling handler <bound method EndpointDiscoveryHandler.handle_retries of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
2022-02-08 17:12:19,035 botocore.discovery [DEBUG] Injecting discovered endpoint: https://ingest-cell1.timestream.eu-central-1.amazonaws.com
2022-02-08 17:12:19,035 botocore.hooks [DEBUG] Event request-created.timestream-write.WriteRecords: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x1ba455520>>
2022-02-08 17:12:19,036 botocore.hooks [DEBUG] Event choose-signer.timestream-write.WriteRecords: calling handler <function set_operation_specific_signer at 0x10cb66ca0>
2022-02-08 17:12:19,036 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-02-08 17:12:19,037 botocore.auth [DEBUG] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:ingest-cell1.timestream.eu-central-1.amazonaws.com
x-amz-date:20220208T161219Z
x-amz-target:Timestream_20181101.WriteRecords

content-type;host;x-amz-date;x-amz-target
be666034f1f56562acb0c69508f237f5556c8cf93771d6a13cabefee515fba3d
2022-02-08 17:12:19,037 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20220208T161219Z
20220208/eu-central-1/timestream/aws4_request
2022-02-08 17:12:19,038 botocore.auth [DEBUG] Signature:
2022-02-08 17:12:19,038 botocore.hooks [DEBUG] Event request-created.timestream-write.WriteRecords: calling handler <function add_retry_headers at 0x10cb6dd30>
2022-02-08 17:12:19,038 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ingest-cell1.timestream.eu-central-1.amazonaws.com, headers={'X-Amz-Target': b'Timestream_20181101.WriteRecords', 'Content-Type': b'application/x-amz-json-1.0', 'User-Agent': b'Boto3/1.20.50 Python/3.9.10 Darwin/21.3.0 Botocore/1.23.50', 'X-Amz-Date': b'20220208T161219Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=/eu-central-1/timestream/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=', 'amz-sdk-invocation-id': b', 'amz-sdk-request': b'attempt=1', 'Content-Length': '478'}>
2022-02-08 17:12:19,039 botocore.httpsession [DEBUG] Certificate path: /usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/certifi/cacert.pem
2022-02-08 17:12:19,040 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): ingest-cell1.timestream.eu-central-1.amazonaws.com:443
2022-02-08 17:12:19,201 urllib3.connectionpool [DEBUG] https://ingest-cell1.timestream.eu-central-1.amazonaws.com:443 "POST / HTTP/1.1" 400 123
2022-02-08 17:12:19,203 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'OF7ZHL7QMX4UL6RNBEOHDJGL6U', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '123', 'Date': 'Tue, 08 Feb 2022 16:12:19 GMT', 'Connection': 'close'}
2022-02-08 17:12:19,203 botocore.parsers [DEBUG] Response body:
b'{"__type":"com.amazonaws.timestream.v20181101#ValidationException","Message":"TIMESTAMP is an invalid measure value type."}'
2022-02-08 17:12:19,204 botocore.parsers [DEBUG] Response headers: {'x-amzn-RequestId': 'OF7ZHL7QMX4UL6RNBEOHDJGL6U', 'Content-Type': 'application/x-amz-json-1.0', 'Content-Length': '123', 'Date': 'Tue, 08 Feb 2022 16:12:19 GMT', 'Connection': 'close'}
2022-02-08 17:12:19,204 botocore.parsers [DEBUG] Response body:
b'{"__type":"com.amazonaws.timestream.v20181101#ValidationException","Message":"TIMESTAMP is an invalid measure value type."}'
2022-02-08 17:12:19,205 botocore.hooks [DEBUG] Event needs-retry.timestream-write.WriteRecords: calling handler <botocore.retryhandler.RetryHandler object at 0x1ba46f160>
2022-02-08 17:12:19,206 botocore.retryhandler [DEBUG] No retry needed.
2022-02-08 17:12:19,206 botocore.hooks [DEBUG] Event needs-retry.timestream-write.WriteRecords: calling handler <bound method EndpointDiscoveryHandler.handle_retries of <botocore.discovery.EndpointDiscoveryHandler object at 0x1ba46f250>>
Traceback (most recent call last):

  File "/var/folders/bh/vqvnzrw12913pc_4ny0bxyfh0000gn/T/ipykernel_35022/3734318343.py", line 1, in <module>
    t_client.write_records(DatabaseName="db",

  File "/usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/usr/local/Caskroom/miniforge/base/envs/env_name/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)

ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type.

BastianZim avatar Feb 05 '22 16:02 BastianZim

Hi @BastianZim,

It looks like this type was added in botocore==1.23.13 here. You have a very recent version of boto3 though, so I'm not sure. I would need to see full debug logs, but we do not have any way to accept private messages. You can redact any sensitive information so they can be posted here, or you would need to open an AWS support case to have private logs reviewed further.

Thanks!

kdaily avatar Feb 08 '22 00:02 kdaily

Hi @kdaily

thanks for the quick reply. My botocore is 1.23.50 so I don't think that's the cause but I have uploaded the redacted log output above. Please let me know if you need anything else!

Thanks!

BastianZim avatar Feb 08 '22 16:02 BastianZim

Thanks @BastianZim that's what I needed, I can see the full request and response. I might need to check in with the Timestream team to go further, will let you know what I learn.

kdaily avatar Feb 08 '22 18:02 kdaily

Hi @kdaily just wanted to quickly ask if you have any news here? Just designing some infrastructure and deciding if I should include Timestream or not.

BastianZim avatar Feb 22 '22 15:02 BastianZim

Hi @BastianZim, You will probably need to tweak your input a little.

common_attributes={'Dimensions': [{'Name': 'name', 'Value': '0928.9274'},
                                  {'Name': 'currency', 'Value': 'USD'},
                                  {'Name': 'type', 'Value': 'cash'}], 
                   'Time': '1647547946000000001', 
                   'TimeUnit': 'NANOSECONDS'}

records_list = [
    {'MeasureName': 'SomeMultiMeasureName',
     'MeasureValueType': 'MULTI',
     'MeasureValues': [
         {'Name': 'Test date', 'Type': 'TIMESTAMP', 'Value': '1647547946000000000'},
         {'Name': 'Test value', 'Type': 'DOUBLE', 'Value': '184532.6961294596385'}
     ]
    }
]

Notice the change in the TimeUnit in common_attributes which is used to derive TimeUnit for the record. Let me know if this resolves the issue.

lohithshetty avatar Mar 17 '22 20:03 lohithshetty

Hi @lohithshetty thanks for the comment and example. Unfortunately, I am still getting the same error. It seems like TIMESTAMP is just not accepted as a measure type, no matter what unit.

BastianZim avatar Mar 18 '22 16:03 BastianZim

Hi @BastianZim Just wanted to clarify that Type 'TIMESTAMP' is only supported within measure values of type 'MULTI' https://docs.aws.amazon.com/timestream/latest/developerguide/writes.html#writes.writing-data-multi-measure for example

[{'MeasureName': 'Test date', 'MeasureValueType': 'TIMESTAMP', 'MeasureValue': '1643987700000000000'}, {'MeasureName': 'Test value', 'MeasureValueType': 'DOUBLE', 'MeasureValue': '184532.6961294596385'}]

would give you TIMESTAMP is an invalid measure value type error

You will have to use MULTI measure value type to use TIMESTAMP like

[{'MeasureName': 'SomeMultiMeasureName',
     'MeasureValueType': 'MULTI',
     'MeasureValues': [
         {'Name': 'Test date', 'Type': 'TIMESTAMP', 'Value': '1647547946000000000'},
         {'Name': 'Test value', 'Type': 'DOUBLE', 'Value': '184532.6961294596385'}
     ]
    }]

and also specify the time unit you want to use for TIMESTAMP in common attributes like 'TimeUnit': 'NANOSECONDS'

Refer here for data modeling: https://docs.aws.amazon.com/timestream/latest/developerguide/data-modeling.html#data-modeling-measurenamemulti

abhijithsatyakumar avatar Mar 18 '22 18:03 abhijithsatyakumar

Hi @abhijithsatyakumar thanks for your reply and sorry for the late answer, somehow I missed it.

I'm having some small troubles with my dependencies which are unrelated to boto3 so I wasn't able to test this yet but from what I can see it seems to be correct.

Please feel free to remove the "investigating" label and I'll close it once I manage to test it.

BastianZim avatar Mar 30 '22 12:03 BastianZim

Checking in - @BastianZim did you have any update on this issue?

tim-finnigan avatar Nov 22 '22 18:11 tim-finnigan

Oh sorry, completely forgot this. Yes, all works and I'm closing.

BastianZim avatar Nov 22 '22 18:11 BastianZim

[ERROR] ValidationException: An error occurred (ValidationException) when calling the WriteRecords operation: TIMESTAMP is an invalid measure value type. Traceback (most recent call last): File "/var/task/lambda_function.py", line 96, in lambda_handler InsertIntoXXXX(payload, client) File "/var/task/lambda_function.py", line 131, in InsertIntoXXXX response = client.write_records( File "/var/runtime/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 960, in _make_api_call raise error_class(parsed_response, operation_name)

Getting this error running code in a lambda function, issue does not seem to be fixed.

LukeRens3 avatar Feb 08 '24 21:02 LukeRens3