How to record and query total requests (flask_http_request_total) PER endpoint/url_rule etc?

[dempstert]

Problem

As per the title, I don't know how to see total requests per endpoint.

Additional Note about uncertainty

I am not sure if the problem is my prometheus queries, or my configuration of flask prometheus itself. I believe the issue is with the configuration.

Question

How to configure prometheus_flask_exporter so that I can record flask_http_request_total per url_rule?

flask_http_request_total seems to group by response code only, but I want to record it on a url_rule bases

flask_http_request_duration_seconds_bucket and flask_http_request_duration_seconds_count and flask_http_request_duration_seconds_sum is grouped by url_rule by default.

What I've Tried

Below is my configuration. I have added default_labels but it doesn't help.

    metrics = GunicornPrometheusMetrics.for_app_factory(
        group_by="url_rule",
        excluded_paths=[
            "/js/*",
            "/css/*",
            "/images/*",
            "/fonts/*",
        ],
        default_labels={"url_rule": lambda: request.url_rule},
    )
    metrics.init_app(app)

[rycus86]

Some ideas that come to mind that you could try:

• aggregate from flask_http_request_duration_seconds_count on Prometheus if these are already tagged with url_rule
• add a default metric as per the example in the README and see if that helps:

metrics.register_default(
    metrics.counter(
        'by_url_rule_counter', 'Request count by URL rule',
        labels={'url_rule': lambda: request.url_rule}
    )
)

[dempstert]

Thanks @rycus86 but I'm afraid I'm not having any luck with that.

Seems you can't call make_default when using an app factory pattern.

Also, I believe that calling init_app calls the register_default method.

What I tried

    metrics = GunicornPrometheusMetrics.for_app_factory(
        group_by="url_rule",
        excluded_paths=[
            "/js/*",
            "/css/*",
            "/images/*",
            "/fonts/*",
        ],
        # default_labels={"url_rule": lambda: request.url_rule},
    )
    counter = metrics.counter(
        "by_url_rule_counter", "Request count by URL rule", labels={"url_rule": lambda: request.url_rule}
    )
    metrics.register_default(counter)

Traceback

  File "/app/prosebit/app.py", line 86, in create_celery_app
    app = app or create_app()
                 ^^^^^^^^^^^^
  File "/app/prosebit/app.py", line 245, in create_app
    metrics.register_default(counter)
  File "/usr/local/lib/python3.11/site-packages/prometheus_flask_exporter/__init__.py", line 563, in register_default
    for endpoint, view_func in app.view_functions.items():
                               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/local.py", line 316, in __get__
    obj = instance._get_current_object()  # type: ignore[misc]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/werkzeug/local.py", line 509, in _get_current_object
    raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

[rycus86]

Would you be able to set up a full minimal example we could test this on? Based on that we should be able to figure out how to support what you're after.

[AndreyNautilus]

I experience the same issue.

Minimal example

I use app-factory approach and I want to make the metrics work locally first before dropping everything into k8s.

__init__.py file

from flask import Flask

from blueprint import backend_bp
from metrics import metrics


def create_app():
    app = Flask(__name__)

    app.register_blueprint(backend_bp)

    # FAILURE: this casues "RuntimeError: Working outside of application context." exception on startup
    # metrics.register_default(
    #     metrics.counter(
    #         'by_path_counter', 'Request count by request paths',
    #         labels={'path': lambda: request.path}
    #     )
    # )
    metrics.init_app(app)

    return app

blueprint.py file:

from flask import Blueprint
from metrics import metrics

backend_bp = Blueprint("backend", __name__)

by_path_counter = metrics.counter(
    'by_path_counter', 'Request count by request paths',
    labels={'path': lambda: request.path}
)


@by_path_counter
@backend_bp.route("/", methods=("GET",))
def root():
    return "root"


@by_path_counter
@backend_bp.route("/info", methods=("GET",))
def info():
    return "info"


@by_path_counter
@backend_bp.route("/custom", methods=("GET",))
def custom():
    return "custom"

metrics.py file:

from prometheus_flask_exporter import PrometheusMetrics

metrics = PrometheusMetrics.for_app_factory()

Python 3.12.1 and dependencies versions (I'm on Windows):

Flask                     3.0.3
prometheus_client         0.21.1
prometheus_flask_exporter 0.23.1

The command: python -m flask --app=. run --port=8001.

If I just run this code, then all endpoints work (localhost:8001/, localhost:8001/info, localhost:8001/custom and localhost:8001/metrics). I make a few requests and inspect the metrics:

• by_path_counter_total appears in comments only:

  ...
  # HELP by_path_counter_total Request count by request paths
  # TYPE by_path_counter_total counter
  # HELP flask_exporter_info Information about the Prometheus Flask exporter
  # TYPE flask_exporter_info gauge
  flask_exporter_info{version="0.23.1"} 1.0
  

• flask_http_request_duration_seconds_bucket appears with all requested path labels:

  flask_http_request_duration_seconds_bucket{le="0.005",method="GET",path="/custom",status="200"} 3.0
  ...
  flask_http_request_duration_seconds_bucket{le="0.005",method="GET",path="/info",status="200"} 4.0
  

If I try to use register_default approach (which I would prefer) and uncomment the code block in __init__.py file and drop manually added by_path_counter from backend.py file, then the app fails to start with the following error:

  File "C:\Projects\learn-k8s\app\.venv\Lib\site-packages\prometheus_flask_exporter\__init__.py", line 563, in register_default
    for endpoint, view_func in app.view_functions.items():
                               ^^^^^^^^^^^^^^^^^^
  File "C:\Projects\learn-k8s\app\.venv\Lib\site-packages\werkzeug\local.py", line 318, in __get__
    obj = instance._get_current_object()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Projects\learn-k8s\app\.venv\Lib\site-packages\werkzeug\local.py", line 519, in _get_current_object
    raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

Counting number of requests per path is pretty simple metrics that first came to my mind. I expect it to be simple to implement, but it turned out to be not-so-simple :) Did I do something wrong?

[rycus86]

Thanks a lot for the minimal example @AndreyNautilus ! I was able to reproduce the issue, and the problem seems to be that register_default expects you either already have an app or the current_app is set up Flask already, which is not the case here.

You can simply pass the app instance to the register_default function and that seems to work ok, so in your code:

def create_app():
    app = Flask(__name__)

    app.register_blueprint(backend_bp)

    metrics.register_default(
        metrics.counter(
            'by_path_counter', 'Request count by request paths',
            labels={'path': lambda: request.path}
        ),
        app=app  # this makes it work
    )
    metrics.init_app(app)

    return app

I've also seen that if you use with app.app_context(): around the metrics registration code you get the same effect, but I think the former is clearer.

[AndreyNautilus]

Thanks @rycus86 ! This helps and now everything works.

[ghost]

Hey @rycus86 , how do we fight paths that have some item ID as a parameter? It's okay when it represents some finite list of statuses/enums, etc, but when it is an OrderID in production systems, cardinality grows so much and each route gets its own metric, causing the /metrics response to have dozens of thousands lines of code, putting pressure to the container resources.

[rycus86]

Hello @adnantrumo have you tried the configuration options in https://github.com/rycus86/prometheus_flask_exporter?tab=readme-ov-file#configuration ? You could try either the endpoint or url_rule grouping perhaps instead of the default path ?

[ghost]

Hello @adnantrumo have you tried the configuration options in https://github.com/rycus86/prometheus_flask_exporter?tab=readme-ov-file#configuration ? You could try either the endpoint or url_rule grouping perhaps instead of the default path ?

Yup, it works! Sorry for bothering!