[💡 FEATURE REQUEST]: Filter logs

[devnev]

Plugin

Logger

I have an idea!

In short, add an option to discard logs matching certain properties/patterns.

Motivation:

I have a global loadbalancer set up, which sends health checks to determine if the backend service is healthy. Previously these were handled by an nginx, with access logs disabled for that particular route. I've now changed the loadbalancer so the healthcheck traverses the full stack, all the way into the PHP worker managed by roadrunner. Now roadrunner is logging these requests, causing a lot of noise between logs of actual traffic. It would be great to be able to stop roadrunner from even emitting these logs.

[rustatian]

Hey @devnev 👋🏻 Could you please share your RR configuration (.rr.yaml)?

[devnev]

@rustatian I'm probably going to have to redact some of it, is there any particular bit you're looking for?

[rustatian]

Yeah, logs section

[devnev]

Logs section is

logs:
  encoding: json
  level: info
  mode: production

[rustatian]

I think that you may simply use channels. info is needed only for the worker logs (if you need them), for the other plugins you may error log level.

[devnev]

I don't think that works for me as I want to have the http logs in general but filter out certain logs (namely successful healthcheck requests)

[rustatian]

Yeah, in that case it makes sense.

[devnev]

I'm actually having the same (or actually more of an issue, same cause but harder to mitigate) with metrics. My RR metrics now include the healthcheck traffic, which makes them less useful.

Would it be possible to run two http server in RR on different ports, with separable logs and metrics? Or otherwise have the status plugin direct healthchecks to the servers? The definiton of ready as currently provided by the status plugin is not particularly useful, in fact I even consider it actively harmful.

[rustatian]

The status plugin reports RR metrics directly to the Prometheus server. In connection with the Grafana you may filter out not needed data and draw only useful for you.

[devnev]

I'm using the status plugin, but the exported metrics don't (and can't) include the HTTP URI that would be needed to filter them.

[rustatian]

Not sure I'm following. The status plugin works by the pull model. It does not send any metrics, metrics are pulled from it. Usually by the local exporter or configured prometheus server. These metrics contains only runtime/workers metrics, not logs.

[devnev]

I'm sending healthcheck requests into the actual worker, to make sure the worker is able to process requests. Because these are http requests received by roadrunner and passed into the worker, these requests appear in logs, and are counted in metrics. I would like to have logs and metrics that exclude these healthcheck requests. For logs, I can post-process the logs to filter out the healtcheck URI. For metrics though, there is no way to exclude the healthcheck requests, as they are aggregated into the request counter and latency histogram metrics without any way of distinguishing them.

And my suggestion is to follow one of two possible approaches:

• Allow creating more http listeners in the http plugin, with a label to distinguish them in the metrics
• Allow the status plugin to forward healthcheck requests to the servers as part of its healthcheck
  • This might be a server plugin feature rather than a status plugin feature?
  • There also need to be two separate types of healthcheck, one for liveness and one for readiness, not sure if RR is handling this at all right now.