LeonD9

@bogdandrutu For balancing traffic you mean using headless service? Regarding rate limiting i would like to be able to throw spans from specific clients that sent above of 1k spans...

> When using gRPC (port 4317), you can use headless services plus client-side load balancing to achieve decent load balancing if you have enough clients. Note that gRPC has a...

Is there something similar for http connections as well? or is it only for gRPC?

@jpkrohling Regarding rate limiting i would like to be able to throw spans from specific clients that sent above of 1k spans per second for example, is it possible with...

@joe-elliott The spans i use for testing are from https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/tracegen so they are pretty small. I tried moving tempo to specific nodes without any other pods and it seems it...

I checked both metrics you suggested before the memory spikes and this are the results: tempo_distributor_ingester_append_failures_total  tempo_distributor_forwarder_queue_length  tempo_distributor_metrics_generator_pushes_failures_total  After that i changed remote_timeout to 100ms and metrics_generator_forwarder_queue_size...

@joe-elliott What do you mean by RF3? Also if the distributor fail writing to the ingester to they have retry function to avoid losing spans? or how can i know...

@patrick-stephens You are correct my mistake regarding the `INPUT` stdout i wrote it incorrectly in the description. I tried using it as you mentioned, this is my fluent-bit.conf file: ```...

Changing the configuration to yaml configuration from the classic configuration helped fix the include issue for me: ``` includes: - ../config/*.yaml ```

@ryn9 removed `type_name` and `include_tag_key` didn't help i still continue to get the following error: ``` 2022-03-28 14:28:44 +0000 [warn]: #0 [opensearch] failed to flush the buffer. retry_times=5 next_retry_time=2022-03-28 14:29:00...