server icon indicating copy to clipboard operation
server copied to clipboard

Increase the metrics's verbose logs level to 2?

Open so2bin opened this issue 1 year ago • 2 comments

Is your feature request related to a problem? Please describe.

  • Normally, we would like to set log verbose=1 for printing the request logs to stdout, like the following image: image
  • But now all the request log is also with log verbose=1 include the metrics api, as showing in the above image.
  • In my k8s cluster, the metrics is calling by the promethues prob that is created by a ServiceMonitor. Now i was troubled by too much many metrics request logs:
I1222 03:39:44.299760 1 http_server.cc:4384] HTTP request: 0 /v2/health/live
I1222 03:39:45.745286 1 http_server.cc:224] HTTP request: 0 /metrics
I1222 03:39:45.939855 1 http_server.cc:224] HTTP request: 0 /metrics
I1222 03:39:46.494887 1 http_server.cc:224] HTTP request: 0 /metrics
I1222 03:39:46.691388 1 http_server.cc:224] HTTP request: 0 /metrics

Describe the solution you'd like

  • As show in the https://github.com/triton-inference-server/server/blob/705a673410ec827c9fb9233725343277af201e2a/src/http_server.cc#L236
    the metrics's log verbose level is 1, so is it reasonable to increase the log level to 2 or higher? Beause these metrics logs is less important in the production environment, but the normally inference request logs is more important.

so2bin avatar Dec 22 '23 03:12 so2bin

I think this is a reasonable ask. @GuanLuo / @rmccorm4 any thoughts? My only concern is that this might be a breaking change if the user relies on these logs to be printed with LOG_VERBOSE(1).

Tabrizian avatar Jan 12 '24 15:01 Tabrizian

CC: @GuanLuo @rmccorm4 @nnshah1

Our logs have gotten more verbose over time. It may be worth discussing what Triton wants in the short- and long-term, so that we do not introduce breaking changes (especially multiple times).

dyastremsky avatar Feb 20 '24 18:02 dyastremsky