Kong prometheus plugin does not record 404 response codes from proxy

[surenraju-careem]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

3.6.1

Current Behavior

When we enable the Prometheus plugin globally, and the proxy returns 404 response codes, the Prometheus plugin does not log the 404 status codes.

This gives us no way to monitor for 404 errors that could be caused by scenarios such as invalid configurations pushed to Kong, resulting in the Kong proxy rejecting all requests.

Expected Behavior

Prometheus metric endpoint records 404 response codes

Steps To Reproduce

Declarative config(declarative/kong.yaml):

_transform: true

services:
  - host: httpbin.org
    name: test-service
    port: 443
    protocol: https
    routes:
      - name: get-route
        paths:
          - /get
        strip_path: false
      - name: post-route
        paths:
          - /post
        methods:
          - POST
        strip_path: false
  - host: api.chucknorris.io
    name: jokes-service
    port: 443
    protocol: https
    routes:
      - name: jokes
        paths:
          - /jokes/random
        strip_path: false
plugins:
  - name: prometheus
    config:
      status_code_metrics: true
      latency_metrics: true
      bandwidth_metrics: true
      upstream_health_metrics: true

Docker Compose:

networks:
  kong-net:
    external: false

services:
  kong:
    image: docker.io/library/kong:3.3.1-ubuntu
    environment:
      KONG_NGINX_WORKER_PROCESSES: 1
      KONG_LOG_LEVEL: "debug"
      KONG_DATABASE: "off"
      KONG_DECLARATIVE_CONFIG: /kong/declarative/kong.yml
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
      KONG_ADMIN_GUI_URL: http://localhost:8002
      KONG_LICENSE_DATA: ""
      KONG_PLUGINS: bundled
    networks:
      - kong-net
    ports:
      - 8000:8000
      - 8443:8443
      - 8001:8001
      - 8444:8444
      - 8002:8002
      - 8445:8445
      - 8003:8003
      - 8004:8004
    volumes:
      - ./declarative:/kong/declarative/

Generate some random traffic http://localhost:8000/random that makes kong return 404, the hit http://localhost:8001/metrics, you will see no 404 reported by prometheus plugin.

[samugi]

hello @surenraju-careem , thank you for opening this issue.

I believe the behavior you describe is expected, some metrics are intentionally not incremented when no route/service is matched, including the http_requests_total, which are intended to count requests by status code "returned by upstream services" as documented.

Changing this behavior would be considered a feature, which we may consider for implementation based on demand and availability. If you would like to open a pull request to support this we would be more than happy to review it.

Let me know if you have any doubts, thank you!

[github-actions[bot]]

This issue is marked as stale because it has been open for 14 days with no activity.

[github-actions[bot]]

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team