grafana
[Tempo-distributed] - Unclear override configuation for tempo-distributed
Describe the bug I am trying to increase the rate limits in tempo, deployed using helm and the chart tempo-distributed and has run into issues understanding the documentation.
To Reproduce
Steps to reproduce the behavior:
Deployed using command:
helm install tempo-distributed grafana/tempo-distributed --namespace tempo --create-namespace --values distributed.yaml
the value file "distributed.yaml" content:
storage:
trace:
backend: s3
s3:
bucket: "tempo-traces"
endpoint: minio.minio.svc.cluster.local:80
access_key: tempo
secret_key: REPLACE
forcepathstyle: true
insecure: true
global_overrides:
per_tenant_override_config: /runtime-config/overrides.yaml
overrides: |-
overrides:
defaults:
ingestion:
burst_size_bytes: 20000000 # 20MB
rate_limit_bytes: 20000000 # 20MB
max_traces_per_user: 10000
ingestion_burst_size_bytes: 20000000 # 20MB
ingestion_rate_limit_bytes: 20000000 # 20MB
global:
max_bytes_per_trace: 20000000 # 20MB
"*":
ingestion:
burst_size_bytes: 20000000 # 20MB
rate_limit_bytes: 20000000 # 20MB
max_traces_per_user: 10000
ingestion_burst_size_bytes: 20000000 # 20MB
ingestion_rate_limit_bytes: 20000000 # 20MB
global:
max_bytes_per_trace: 20000000 # 20MB
ingester:
replicas: 1
traces:
otlp:
http:
enabled: true
grpc:
enabled: true
I have seen multiple errors: the current is: "field ingestion not found in type overrides.LegacyOverrides"
The question is
- How should "global_overrides" be used ?
- what is the correct format for "overrides:" ?
I have been trying to use this as a guide https://grafana.com/docs/tempo/latest/configuration/#runtime-overrides
Expected behavior I am sure this is me not fully understanding the documentation. A single example for a value file for my version (1.9.9) would be much appreciated, where typical rate limit configuation is done. It might be related to https://github.com/grafana/helm-charts/issues/2802
Environment:
- Infrastructure: kubernetes
- Deployment tool: helm
- helm Chart version: 1.9.9
- Tempo version: 2.4.1
Link to issue in tempo repo: https://github.com/grafana/helm-charts/issues/3171
I came here to file the same ticket. By examining the tempo and chart source, I was able to puzzle out the correct configuration. (And it has nothing to do with what is documented.)
Put something like this in your overrides.yaml file.
tempo:
structuredConfig:
overrides:
defaults:
ingestion:
rate_limit_bytes: 40000000
burst_size_bytes: 50000000
max_traces_per_user: 30000
global:
max_bytes_per_trace: 8000000
I came here to file the same ticket. By examining the tempo and chart source, I was able to puzzle out the correct configuration. (And it has nothing to do with what is documented.)
Put something like this in your overrides.yaml file.
tempo: structuredConfig: overrides: defaults: ingestion: rate_limit_bytes: 40000000 burst_size_bytes: 50000000 max_traces_per_user: 30000 global: max_bytes_per_trace: 8000000
This fails for me with
failed parsing config: failed to parse configFile /conf/tempo.yaml: yaml: unmarshal errors:
line 76: field defaults not found in type overrides.legacyConfig
Chart: 1.9.11 App Version: 2.4.2
Chart
I'm using chart 1.9.10, but don't see anything in the diff that would matter. :shrug:
If you want to workaround this Helm Chart issue, just use as follows:
global_overrides:
defaults:
ingestion:
rate_limit_bytes: 32000000 # 32MB
burst_size_bytes: 48000000 # 48MB
max_traces_per_user: 50000
Here I explain a little bit further why the issue is happening.
global_overrides: defaults: ingestion: rate_limit_bytes: 32000000 # 32MB burst_size_bytes: 48000000 # 48MB max_traces_per_user: 50000
This one not working for me...
field defaults not found in type overrides.legacyConfig
Hi, @buker . You probably have to figure out why your configs are using the legacyConfig.
The defaults block is from the current Config format.
For the record, check Tempo code for:
Hope that helps figuring things out.
Hi there! I noticed that we had a bunch of issues open for the overrides settings for the tempo-distributed Helm. chart. We have two updates that may address the issue that you're having:
- https://github.com/grafana/helm-charts/pull/3468
- https://github.com/grafana/tempo/pull/4415
Please let me know if this addresses the issue.
Hi, I have already closed the related issue. I will also go ahead and close this. Thanks for your help, and the new documentation is making it more clear.
For ppl with similar issues, I explain what I did to make it work. We went from chart 1.32.7 to 1.39.3 and according to https://github.com/grafana/helm-charts/tree/main/charts/tempo-distributed, from 1.33.0 onwards there is a change regarding the global_overrides. I did rename the global_overrides to overrides, then I got the 'field defaults not found in type overrides.legacyConfig' issue.
Finally I created a dummy configuration overrides file in /tmp/overrides.yaml:
overrides:
defaults:
metrics_generator:
processors:
- service-graphs
- span-metrics
- local-blocks
and then I used the tempo-cli tool to generate the new configuration:
docker run --rm -v /tmp:/runtime-config grafana/tempo-cli migrate overrides-config /runtime-config/overrides.yaml
The I put this in my values.yaml and everything was sunny again:
overrides:
defaults:
ingestion:
rate_strategy: local
rate_limit_bytes: 15000000
burst_size_bytes: 20000000
max_traces_per_user: 10000
read:
max_bytes_per_tag_values_query: 1000000
metrics_generator:
processors:
- span-metrics
- local-blocks
- service-graphs
generate_native_histograms: classic
ingestion_time_range_slack: 0s
global:
max_bytes_per_trace: 5000000
