watermarkpodautoscaler
watermarkpodautoscaler copied to clipboard
unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Invalid metric (from backend)
{"level":"info","ts":1624011129.786673,"logger":"controllers.WatermarkPodAutoscaler","msg":"Failed to compute desired number of replicas based on listed metrics.","watermarkpodautoscaler":"dev/myapp","reference":"Deployment/dev/myapp","error":"failed to get external metric kubernetes.cpu.usage: unable to get external metric dev/kubernetes.cpu.usage/&LabelSelector{MatchLabels:map[string]string{app: myapp,release: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Invalid metric (from backend), query: avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)"} {"level":"info","ts":1624011144.793971,"logger":"controllers.WatermarkPodAutoscaler","msg":"Target deploy","watermarkpodautoscaler":"dev/myapp","replicas":2} {"level":"info","ts":1624011144.7941537,"logger":"controllers.WatermarkPodAutoscaler","msg":"getReadyPodsCount","watermarkpodautoscaler":"dev/myapp","full podList length":2,"toleratedAsReadyPodCount":2,"incorrectly targeted pods":0} {"level":"info","ts":1624011144.8295028,"logger":"controllers.WatermarkPodAutoscaler","msg":"Failed to compute desired number of replicas based on listed metrics.","watermarkpodautoscaler":"dev/myapp","reference":"Deployment/dev/myapp","error":"failed to get external metric kubernetes.cpu.usage: unable to get external metric dev/kubernetes.cpu.usage/&LabelSelector{MatchLabels:map[string]string{app: myapp,release: myapp,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: Internal error occurred: DatadogMetric is invalid, err: Invalid metric (from backend), query: avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)"} {"level":"info","ts":1624011159.8374639,"logger":"controllers.WatermarkPodAutoscaler","msg":"Target deploy","watermarkpodautoscaler":"dev/myapp","replicas":2}
Ask for metric (without tags)
% kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/dev/kubernetes.cpu.usage | jq ." Error from server (InternalError): Internal error occurred: DatadogMetric not found for metric name: kubernetes.cpu.usage | jq ., datadogmetricid: datadog/dcaautogen-646a73ad876299907eb8035a2fa8e2b60ac832
Impersonate and ask for any metric
% kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/dev/metric --as system:serviceaccount:dev:watermarkpodautoscaler" Error from server (InternalError): Internal error occurred: DatadogMetric not found for metric name: metric --as system:serviceaccount:dev:watermarkpodautoscaler, datadogmetricid: datadog/dcaautogen-98eae079d0f4a80135d6f4b6f9762cea878b97
What did i wrong? How can i general find out which metrics and which tags for it are available?
Hello @cforce,
It normally means that no data is available for avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)"
, you can put this query in a Notebook to visualize the data used for autoscaling.
If you'd like to see what the API returns, you can use a small Python script using our Datadog client:
import time
import json
from datadog import initialize, api
options = {
'api_key': '<api_key>',
'app_key': '<app_key>'
}
initialize(**options)
print(json.dumps(api.Metric.query(start=int(time.time()) - 10, end=int(time.time()), query='avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)'), sort_keys=True, indent=4))
I used this modified script:
import time
import json
import requests
import sys
import json
import argparse
from datadog import initialize, api
print('Executing script file is:', str(sys.argv[0]))
print('The arguments are:', str(sys.argv))
parser = argparse.ArgumentParser()
parser.add_argument("--apikey", help="Provide the DataDog ApiKey.", required=True)
parser.add_argument("--appkey", help="Provide the DataDog AppKey.", required=True)
parser.add_argument("--query", help="Provide the metric query to check, e.g 'avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)'", required=True)
args = parser.parse_args()
apiKey = args.apikey
appKey = args.appkey
query = args.query
options = {
'api_key': apiKey,
'app_key': appKey,
'application_key': appKey
}
initialize(**options)
print(json.dumps(api.Metric.query(start=int(time.time()) - 10, end=int(time.time()), query=query), sort_keys=True, indent=4))
Response:
{
"from_date": 1624114030000,
"group_by": [],
"message": "",
"query": "avg:kubernetes.cpu.usage{app:myapp,release:myapp}.rollup(30)",
"res_type": "time_series",
"resp_version": 1,
"series": [],
"status": "ok",
"times": [],
"to_date": 1624114040000,
"values": []
}
Seems there are no values? So the tags does not fit - how to try out without and howto find out which are available? I tried with
- avg:kubernetes.cpu.usage{service:myapp}.rollup(30) - also values[]
- avg:kubernetes.cpu.usage{}.rollup(30) - failure
- avg:kubernetes.cpu.usage.rollup(30) - failure
- avg:kubernetes.cpu.usage - failure
That's the expected behaviour indeed. About avg:kubernetes.cpu.usage{}.rollup(30)
, querying without any tag is avg:kubernetes.cpu.usage{*}.rollup(30)
.
About checking available tags, you have 2 tools in Datadog to explore metrics/tags:
- Metrics explorer
- Notebooks
Now if you want to have all possible tags for a metric, you can check the Metrics summary (for instance for kubernetes.cpu.usage.total
https://app.datadoghq.com/metric/summary?filter=kubernetes.cpu.&metric=kubernetes.cpu.usage.total)