text-generation-inference

text-generation-inference copied to clipboard

Feature request

The Prometheus builder, use port 9000 by default. This can be changed with e.g builder.with_http_listener(([0, 0, 0, 0], 9001)), however TGI doesn't support such configuration. Please implement options like --prometheus-port and --prometheus-addr (the latter is not important to me).

Motivation

This feature being missing is quite problematic for me, as I'm on a system where I cannot create an isolated network and port 9000 is already in use. This results in the failed to build prometheus recorder from TGI.

Your contribution

I've contributed with specific guidance, and I've made made major contributions to other huggingface projects. However, I'm presently not strong enough in Rust to do this well.

References:

• Current implementation by TGI: https://github.com/huggingface/text-generation-inference/blob/5eec3a8bb6554ebb942bdaae309cfd3080c4e997/router/src/server.rs#L2101C19-L2117
• Mentioned solution https://github.com/metrics-rs/metrics/issues/501#issuecomment-2263784130