bug: Cloudwatch get-metric-data using Expression fails and strangely looks for alternate field MetricStat

[sai-prasad-m]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

Install localstack and awslocal cli and start localstack via Terminal as shown below

$ localstack start

 💻 LocalStack CLI 3.7.2
 👤 Profile: default

[19:15:09] starting LocalStack in Docker mode 🐳                                                                                                                  localstack.py:503
────────────────────────────────────────────────────────────────── LocalStack Runtime Log (press CTRL-C to quit) ──────────────────────────────────────────────────────────────────

LocalStack version: 3.7.3.dev61
LocalStack build date: 2024-09-30
LocalStack build git hash: dc046d99a

Ready.
2024-10-01T17:15:47.413  INFO --- [et.reactor-0] localstack.request.aws     : AWS cloudwatch.ListMetrics => 200


2024-10-01T17:24:12.543  INFO --- [et.reactor-0] localstack.request.aws     : AWS cloudwatch.GetMetricData => 200
2024-10-01T17:26:20.270 ERROR --- [et.reactor-0] l.aws.handlers.logging     : exception during call chain: 'MetricStat'
2024-10-01T17:26:20.271  INFO --- [et.reactor-0] localstack.request.aws     : AWS cloudwatch.GetMetricData => 500 (InternalError)

Populate a metric in to localstack

$ awslocal cloudwatch put-metric-data --namespace "service/my-service" --metric-name "request.count" --dimensions call_type=service_call,request_status=success,service_status=enabling,tenant=/my/qa,registration_country=gb --value 0.2

$ awslocal cloudwatch put-metric-data --namespace "service/my-service" --metric-name "request.count" --dimensions call_type=service_call,request_status=success,service_status=not_enabling,tenant=/my/qa,registration_country=gb --value 0.02

The metrics.json for fetching the inserted metrics is as follows.

[
    {
        "Id": "e1",
        "Expression": "REMOVE_EMPTY(SEARCH(' Namespace=service/my-service MetricName=request.count service_status=enabling OR service_status=not_enabling request_status=success ', 'Sum'))",
        "Label": "e1",
        "Period": 2160
    }
]

$ awslocal cloudwatch get-metric-data --metric-data-queries file:///home/guestdev/metrics.json --start-time 2024-10-01T00:00:0000 --end-time 2024-10-01T23:00:0000

An error occurred (InternalError) when calling the GetMetricData operation (reached max retries: 4): exception while calling cloudwatch.GetMetricData: 'MetricStat'

However, when I try with actual aws cloudwatch, the metrics are returned as expected

Expected Behavior

When using localstack for cloudwatch metrics and fetching them using expressions, the documentation says either Expression or MetricStat ( not both )

However, when i try with actual cloudwatch, the fetch using expression works as intended. But testing the same using localstack, i get an error

An error occurred (InternalError) when calling the GetMetricData operation (reached max retries: 4): exception while calling cloudwatch.GetMetricData: 'MetricStat'

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

localstack start

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

$ awslocal cloudwatch put-metric-data --namespace "service/my-service" --metric-name "request.count" --dimensions call_type=service_call,request_status=success,service_status=enabling,tenant=/my/qa,registration_country=gb --value 0.2

$ awslocal cloudwatch put-metric-data --namespace "service/my-service" --metric-name "request.count" --dimensions call_type=service_call,request_status=success,service_status=not_enabling,tenant=/my/qa,registration_country=gb --value 0.02

The metrics.json for fetching the inserted metrics is as follows.

[
    {
        "Id": "e1",
        "Expression": "REMOVE_EMPTY(SEARCH(' Namespace=service/my-service MetricName=request.count service_status=enabling OR service_status=not_enabling request_status=success ', 'Sum'))",
        "Label": "e1",
        "Period": 2160
    }
]

$ awslocal cloudwatch get-metric-data --metric-data-queries file:///home/guestdev/metrics.json --start-time 2024-10-01T00:00:0000 --end-time 2024-10-01T23:00:0000

An error occurred (InternalError) when calling the GetMetricData operation (reached max retries: 4): exception while calling cloudwatch.GetMetricData: 'MetricStat'

Environment

- OS: Ubuntu 20.04
- LocalStack:
  LocalStack CLI 3.7.2
  LocalStack version: 3.7.3.dev61
  LocalStack build date: 2024-09-30 
  LocalStack build git hash: dc046d99a

Anything else?

No response

[localstack-bot]

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

[steffyP]

Hi @sai-prasad-m,

thanks for reporting. Expressions are not yet supported by LocalStack, it currently only expects MetricStat parameters.

We don't have an estimate when Expressions will be supported.

However, dealing with the input and providing a proper error message would be an improvement here, that would also be clearer on why the query is failing.

/cc @pinzon

This is the stacktrace in LocalStack:

2024-10-02T15:43:05.566  INFO --- [et.reactor-1] localstack.request.aws     : AWS cloudwatch.GetMetricData => 500 (InternalError)
2024-10-02T15:43:07.205 ERROR --- [et.reactor-0] l.aws.handlers.logging     : exception during call chain
Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/service.py", line 113, in __call__
    handler(chain, context, response)
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/handlers/service.py", line 83, in __call__
    skeleton_response = self.skeleton.invoke(context)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 154, in invoke
    return self.dispatch_request(serializer, context, instance)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 168, in dispatch_request
    result = handler(context, instance) or {}
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/aws/skeleton.py", line 118, in __call__
    return self.fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.11/site-packages/localstack/services/cloudwatch/provider_v2.py", line 241, in get_metric_data
    non_unique.append(query["MetricStat"][diff])
                      ~~~~~^^^^^^^^^^^^^^
KeyError: 'MetricStat'

[sai-prasad-m]

@steffyP Thank you for the response. I will await then.

[localstack-bot]

Hello 👋! It looks like this issue hasn’t been active in longer than five months. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.