kong not show kong_http

Is there an existing issue for this?

[X] I have searched the existing issues

Kong version (`$ kong version`)

image: kong:2.8

Current Behavior

helm.sh/chart: kong-2.10.2 image: kong/kubernetes-ingress-controller:2.3

curl -i http://localhost:8100/metrics

HTTP/1.1 200 OK Date: Sat, 02 Jul 2022 16:52:17 GMT Content-Type: text/plain; charset=UTF-8 Transfer-Encoding: chunked Connection: keep-alive Access-Control-Allow-Origin: * X-Kong-Admin-Latency: 0 Server: kong/2.8.1

kong_memory_lua_shared_dict_bytes{shared_dict="kong",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_bytes{shared_dict="kong_cluster_events",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache",kong_subsystem="http"} 802816 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache_2",kong_subsystem="http"} 802816 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache_miss",kong_subsystem="http"} 90112 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache_miss_2",kong_subsystem="http"} 90112 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache",kong_subsystem="http"} 794624 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache_2",kong_subsystem="http"} 794624 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache_miss",kong_subsystem="http"} 86016 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache_miss_2",kong_subsystem="http"} 86016 kong_memory_lua_shared_dict_bytes{shared_dict="kong_healthchecks",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_bytes{shared_dict="kong_locks",kong_subsystem="http"} 61440 kong_memory_lua_shared_dict_bytes{shared_dict="kong_process_events",kong_subsystem="http"} 45056 kong_memory_lua_shared_dict_bytes{shared_dict="kong_rate_limiting_counters",kong_subsystem="http"} 86016 kong_memory_lua_shared_dict_bytes{shared_dict="prometheus_metrics",kong_subsystem="http"} 49152

kong_memory_lua_shared_dict_total_bytes{shared_dict="kong",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_cluster_events",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache_2",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache_miss",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache_miss_2",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache_2",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache_miss",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache_miss_2",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_healthchecks",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_locks",kong_subsystem="http"} 8388608 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_process_events",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_rate_limiting_counters",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="prometheus_metrics",kong_subsystem="http"} 5242880

kong_memory_workers_lua_vms_bytes{pid="1110",kong_subsystem="http"} 75632874 kong_memory_workers_lua_vms_bytes{pid="1111",kong_subsystem="http"} 75635124

kong_nginx_http_current_connections{state="accepted"} 320 kong_nginx_http_current_connections{state="active"} 3 kong_nginx_http_current_connections{state="handled"} 320 kong_nginx_http_current_connections{state="reading"} 0 kong_nginx_http_current_connections{state="total"} 880 kong_nginx_http_current_connections{state="waiting"} 2 kong_nginx_http_current_connections{state="writing"} 1 kong_nginx_metric_errors_total 0 kong_nginx_timers{state="pending"} 6 kong_nginx_timers{state="running"} 0

Expected Behavior

kong_bandwidth{type="egress",service="google",route="google.route-1"} 1277 kong_bandwidth{type="ingress",service="google",route="google.route-1"} 254

kong_nginx_timers{state="running"} 3 kong_nginx_timers{state="pending"} 1

kong_datastore_reachable 1 kong_http_consumer_status{service="s1",route="s1.route-1",code="200",consumer="<CONSUMER_USERNAME>"} 3 kong_http_status{code="301",service="google",route="google.route-1"} 2 kong_latency_bucket{type="kong",service="google",route="google.route-1",le="00001.0"} 1 kong_latency_bucket{type="kong",service="google",route="google.route-1",le="00002.0"} 1 kong_latency_bucket{type="kong",service="google",route="google.route-1",le="+Inf"} 2 kong_latency_bucket{type="request",service="google",route="google.route-1",le="00300.0"} 1 kong_latency_bucket{type="request",service="google",route="google.route-1",le="00400.0"} 1

kong_latency_bucket{type="request",service="google",route="google.route-1",le="+Inf"} 2 kong_latency_bucket{type="upstream",service="google",route="google.route-1",le="00300.0"} 2 kong_latency_bucket{type="upstream",service="google",route="google.route-1",le="00400.0"} 2 kong_latency_bucket{type="upstream",service="google",route="google.route-1",le="+Inf"} 2 kong_latency_count{type="kong",service="google",route="google.route-1"} 2 kong_latency_count{type="request",service="google",route="google.route-1"} 2 kong_latency_count{type="upstream",service="google",route="google.route-1"} 2 kong_latency_sum{type="kong",service="google",route="google.route-1"} 2145 kong_latency_sum{type="request",service="google",route="google.route-1"} 2672 kong_latency_sum{type="upstream",service="google",route="google.route-1"} 527 kong_nginx_http_current_connections{state="accepted"} 8 kong_nginx_http_current_connections{state="active"} 1 kong_nginx_http_current_connections{state="handled"} 8 kong_nginx_http_current_connections{state="reading"} 0 kong_nginx_http_current_connections{state="total"} 8 kong_nginx_http_current_connections{state="waiting"} 0 kong_nginx_http_current_connections{state="writing"} 1 kong_memory_lua_shared_dict_bytes{shared_dict="kong",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong",kong_subsystem="http"} 5242880 kong_memory_workers_lua_vms_bytes{pid="7281",kong_subsystem="http"} 41124353 kong_data_plane_config_hash{node_id="d4e7584e-b2f2-415b-bb68-3b0936f1fde3",hostname="ubuntu-bionic",ip="127.0.0.1"} 1.7158931820287e+38 kong_data_plane_last_seen{node_id="d4e7584e-b2f2-415b-bb68-3b0936f1fde3",hostname="ubuntu-bionic",ip="127.0.0.1"} 1600190275 kong_data_plane_version_compatible{node_id="d4e7584e-b2f2-415b-bb68-3b0936f1fde3",hostname="ubuntu-bionic",ip="127.0.0.1",kong_version="2.4.1"} 1 kong_nginx_metric_errors_total 0

kong_upstream_target_health{upstream="<upstream_name>",target="",address=":",state="healthchecks_off",subsystem="http"} 0 kong_upstream_target_health{upstream="<upstream_name>",target="",address=":",state="healthy",subsystem="http"} 1 kong_upstream_target_health{upstream="<upstream_name>",target="",address=":",state="unhealthy",subsystem="http"} 0 kong_upstream_target_health{upstream="<upstream_name>",target="",address=":",state="dns_error",subsystem="http"} 0 kong_db_entities_total 42 kong_db_entity_count_errors 0

Steps To Reproduce

>helm install mykong kong/kong --namespace kong --create-namespace --values kong.yaml --set ingressController.installCRDs=false

> cat kong.yaml
podAnnotations:
   prometheus.io/scrape: "true" # Ask Prometheus to scrape the
   prometheus.io/port: "8100"   # Kong pods for metrics

-->
https://docs.konghq.com/hub/kong-inc/prometheus/#example-config

Anything else?

apiVersion: configuration.konghq.com/v1 kind: KongClusterPlugin metadata: name: prometheus annotations: kubernetes.io/ingress.class: kong labels: global: "true" plugin: prometheus No response

Jul 02 '22 17:07 mdmartinezpad

@chronolaw Hello :), how nice to write to you, I wanted to ask you a question: If this error is going to be fixed with a fix? Or am I doing some configuration wrong? Thank you

Jul 05 '22 15:07 mdmartinezpad

I'm facing with same issue on K8S cluster Kong version: 2.8.1 plugin: prometheus

Jul 07 '22 08:07 hnhai

@mdmartinezpad I have fixed this problem you should add this config config: per_consumer: "true"

apiVersion: configuration.konghq.com/v1 kind: KongClusterPlugin metadata: name: prometheus annotations: kubernetes.io/ingress.class: kong labels: global: "true" config: per_consumer: "true" plugin: prometheus

Jul 13 '22 02:07 hnhai

Hello @hnhai , how are you, thanks for responding, but I just added and tried again but the kong status code metric still does not appear. Could you please share your additional yamels files to compare with the ones I have? Thank you.

Jul 13 '22 16:07 mdmartinezpad

Sorry @mdmartinezpad about bad code format. This is full my configuration

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: prometheus
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
plugin: prometheus
disabled: false
config:
  per_consumer: true

After add plugin, call some http request to your service and check /metrics

Jul 14 '22 02:07 hnhai

Hello @hnhai, I apply the configuration then I do port-forward 8100 then load url localhost:8100/metrics

HELP kong_datastore_reachable Datastore reachable from Kong, 0 is unreachable TYPE kong_datastore_reachable gauge kong_datastore_reachable 1 HELP kong_memory_lua_shared_dict_bytes Allocated slabs in bytes in a shared_dict TYPE kong_memory_lua_shared_dict_bytes gauge kong_memory_lua_shared_dict_bytes{shared_dict="kong",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_bytes{shared_dict="kong_cluster_events",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache",kong_subsystem="http"} 802816 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache_2",kong_subsystem="http"} 802816 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache_miss",kong_subsystem="http"} 90112 kong_memory_lua_shared_dict_bytes{shared_dict="kong_core_db_cache_miss_2",kong_subsystem="http"} 90112 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache",kong_subsystem="http"} 794624 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache_2",kong_subsystem="http"} 794624 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache_miss",kong_subsystem="http"} 86016 kong_memory_lua_shared_dict_bytes{shared_dict="kong_db_cache_miss_2",kong_subsystem="http"} 86016 kong_memory_lua_shared_dict_bytes{shared_dict="kong_healthchecks",kong_subsystem="http"} 40960 kong_memory_lua_shared_dict_bytes{shared_dict="kong_locks",kong_subsystem="http"} 61440 kong_memory_lua_shared_dict_bytes{shared_dict="kong_process_events",kong_subsystem="http"} 45056 kong_memory_lua_shared_dict_bytes{shared_dict="kong_rate_limiting_counters",kong_subsystem="http"} 86016 kong_memory_lua_shared_dict_bytes{shared_dict="prometheus_metrics",kong_subsystem="http"} 49152 HELP kong_memory_lua_shared_dict_total_bytes Total capacity in bytes of a shared_dict TYPE kong_memory_lua_shared_dict_total_bytes gauge kong_memory_lua_shared_dict_total_bytes{shared_dict="kong",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_cluster_events",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache_2",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache_miss",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_core_db_cache_miss_2",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache_2",kong_subsystem="http"} 134217728 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache_miss",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_db_cache_miss_2",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_healthchecks",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_locks",kong_subsystem="http"} 8388608 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_process_events",kong_subsystem="http"} 5242880 kong_memory_lua_shared_dict_total_bytes{shared_dict="kong_rate_limiting_counters",kong_subsystem="http"} 12582912 kong_memory_lua_shared_dict_total_bytes{shared_dict="prometheus_metrics",kong_subsystem="http"} 5242880 HELP kong_memory_workers_lua_vms_bytes Allocated bytes in worker Lua VM TYPE kong_memory_workers_lua_vms_bytes gauge kong_memory_workers_lua_vms_bytes{pid="1109",kong_subsystem="http"} 75783529 kong_memory_workers_lua_vms_bytes{pid="1110",kong_subsystem="http"} 75789581 HELP kong_nginx_http_current_connections Number of HTTP connections TYPE kong_nginx_http_current_connections gauge kong_nginx_http_current_connections{state="accepted"} 3432 kong_nginx_http_current_connections{state="active"} 4 kong_nginx_http_current_connections{state="handled"} 3432 kong_nginx_http_current_connections{state="reading"} 0 kong_nginx_http_current_connections{state="total"} 9535 kong_nginx_http_current_connections{state="waiting"} 3 kong_nginx_http_current_connections{state="writing"} 1 HELP kong_nginx_metric_errors_total Number of nginx-lua-prometheus errors TYPE kong_nginx_metric_errors_total counter kong_nginx_metric_errors_total 0 HELP kong_nginx_timers Number of nginx timers TYPE kong_nginx_timers gauge kong_nginx_timers{state="pending"} 6 kong_nginx_timers{state="running"} 0

Not show status code kong :(

Jul 15 '22 04:07 mdmartinezpad

@mdmartinezpad did you call any request to your service throught ingress ?

Jul 15 '22 04:07 hnhai

Dear contributor, We're closing this issue as there hasn't been any update to it for a long time. If the issue is still relevant in the latest version, please feel free to reopen it. We're more than happy to revisit it again. Your contribution is greatly appreciated! Please have a look at our pledge to the community for more information. Sincerely, Kong Gateway Team

Oct 11 '23 06:10 StarlightIbuki

not show kong_http_status

Is there an existing issue for this?

Kong version ($ kong version)

Current Behavior

Expected Behavior

Steps To Reproduce

Anything else?

Kong version (`$ kong version`)