Possibility to label or ignore requests in metrics

[jan-stanek]

Describe the feature

Currently there is no possibility to ignore health check metrics in knative metrics. We have multiple knative services with different health check URL's which are called more often than other service endpoints, so the metrics are unusable for us, becase we can't distinguish between real traffic and healthchecks (we use knative as a part of https://github.com/kserve/kserve and want to have just traffic to the model in dashboards).

Related issues: #14581 #15646

Proposed solution: #15653, https://github.com/knative/docs/pull/6177 (health check request have special header and these requests are ignored in metrics), another possibility is to add some label to metrics and filer them in tools like Grafana.

I hope that it's pretty easy and no google doc is needed.

@evankanderson

[dprotaso]

I hope that it's pretty easy and no google doc is needed.

I'd prefer a more flushed out proposal for this feature see: https://github.com/knative/community/blob/main/mechanics/FEATURE-TRACKS.md

The goal of the proposal is to surface questions which Evan has been doing in GitHub Issues: https://github.com/knative/docs/pull/6177#discussion_r1950056755

Part of this is defining what happens when revisions are scaled to zero etc

[0xV0YD]

@dprotaso @jan-stanek can i work on this issue ?

[jan-stanek]

Sure, I haven't started yet. We've found out that health checks also prevents scaling to zero, so we stopped using them

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[evankanderson]

/remove-lifecycle stale