Clarify how burst limits config works in Promtail

[salvacorts]

The documentation for Promtail's readline_burst limit is not clear enough about when bursts are allowed. It currently states the following:

# The cap in the quantity of burst lines that this instance of Promtail may push
# to Loki.
[readline_burst: <int> | default = 10000]

We use golang.org/x/time/rate under the hood. Where b is readline_burst and r is readline_rate. According to it’s documentation:

… It implements a “token bucket” of size b, initially full and refilled at rate r tokens per second. Informally, in any large enough time interval, the Limiter limits the rate to r tokens per second, with a maximum burst size of b events … See https://en.wikipedia.org/wiki/Token_bucket for more about token buckets.

In other words, the average rate will be readline_rate, but bursts up to a size of readline_burst are allowed from time to time.

The burst limit is enforced here: https://github.com/golang/time/blob/579cf78fd858857c0d766e0d63eb2b0ccf29f436/rate/rate.go#L396-L398

For example, If readline_rate is 100 and readline_burst is 200, and they send 150 items exactly once per second, then the first 150 will go through, and thereafter, they will be able to send up to 100 items per second. If they stopped sending for 2 seconds, they could send 150 in one go again and subsequent sends will be limited to 100 items per second. In any case, if the rate is above readline_burst, for example 250, the 50 items over the bust size will be dropped.

Note that there are other burst limits within Loki that maybe benefit from the clarification of Promtail's readline_burst. Here is a list:

• ingestion_burst_size_mb: The documentation for this limit config is more clear than Promtail's readline_burst, but there may be room for improvement.
• per_stream_rate_limit_burst
• rate_limit_burst

[KMiller-Grafana]

Slack discussion on the definitions and how the rate/burst work: https://raintank-corp.slack.com/archives/C029V4SSS9L/p1658750411188449.

[GrafanaWriter]

@KMiller-Grafana to identify what's completed and what is still in need of development, splitting the issue in two.

[GrafanaWriter]

@JStickler - can you please review with the team to prioritize?