server
server copied to clipboard
Increase the metrics's verbose logs level to 2?
Is your feature request related to a problem? Please describe.
- Normally, we would like to set log verbose=1 for printing the request logs to stdout, like the following image:
- But now all the request log is also with log verbose=1 include the metrics api, as showing in the above image.
- In my k8s cluster, the metrics is calling by the promethues prob that is created by a
ServiceMonitor. Now i was troubled by too much many metrics request logs:
I1222 03:39:44.299760 1 http_server.cc:4384] HTTP request: 0 /v2/health/live
I1222 03:39:45.745286 1 http_server.cc:224] HTTP request: 0 /metrics
I1222 03:39:45.939855 1 http_server.cc:224] HTTP request: 0 /metrics
I1222 03:39:46.494887 1 http_server.cc:224] HTTP request: 0 /metrics
I1222 03:39:46.691388 1 http_server.cc:224] HTTP request: 0 /metrics
Describe the solution you'd like
- As show in the
https://github.com/triton-inference-server/server/blob/705a673410ec827c9fb9233725343277af201e2a/src/http_server.cc#L236
the metrics's log verbose level is 1, so is it reasonable to increase the log level to 2 or higher? Beause these metrics logs is less important in the production environment, but the normally inference request logs is more important.
I think this is a reasonable ask. @GuanLuo / @rmccorm4 any thoughts? My only concern is that this might be a breaking change if the user relies on these logs to be printed with LOG_VERBOSE(1).
CC: @GuanLuo @rmccorm4 @nnshah1
Our logs have gotten more verbose over time. It may be worth discussing what Triton wants in the short- and long-term, so that we do not introduce breaking changes (especially multiple times).