Document how to deploy Tempo to ingest traces from multiple clusters

[kvrhdn]

It's possible to ingest traces from multiple clusters into a single big Tempo instance. I believe this is a fairly common deployment setup, so we should try to describe this in our docs.

We should describe:

• how to setup ingress (set up a load balancer, add a gateway for authentication and routing)
• how to collect traces in each cluster and forward them to Tempo (i.e. set up a bunch of Grafana Agents with a service in front, remote write to Tempo)

Also good to mention: load balancing GRPC streams has some tricky behaviour, see https://grpc.io/blog/grpc-load-balancing/

Question about this setup in the community slack: https://grafana.slack.com/archives/C01D981PEE5/p1632219533109300

[BitProcessor]

I'm using the following:

  chart:
    repository: https://grafana.github.io/helm-charts
    name: tempo-distributed
    version: 0.9.14

and I added an Ingress object like this:

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: prod-tempo
  name: tempo
  annotations:
    kubernetes.io/ingress.class: nginx-internal
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: GRPC    
spec:
  tls:
  - hosts:
    - otlp-grpc.tempo.internal.<hidden>.io
    secretName: le-prod-tempo
  rules:
  - host: otlp-grpc.tempo.internal.<hidden>.io
    http:
      paths:
      - path: /
        backend:
          serviceName: tempo-tempo-distributed-distributor
          servicePort: 4317

Seems to be working just fine for now. This uses the opensource nginx-ingress in the back + AWS Classic ELB (on private subnet shared between multiple clusters)

[jacek-czernik]

@BitProcessor do you have any custom config on the AWS Classic LB to make it work? as per AWS docs , the load balancer you are using does not support gRPC protocol.

[BitProcessor]

@jacek-czernik LB is in TCP mode (which is supported)

[GrafanaWriter]

Tied to getting started content; related to GET updates that covers multi-tenancy.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[BMonsalvatge]

Would still love to see this doc created for this type of setup, as we're trying to do something similar.

[mehta-ankit]

I'm using the following:

  chart:
    repository: https://grafana.github.io/helm-charts
    name: tempo-distributed
    version: 0.9.14

and I added an Ingress object like this:

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: prod-tempo
  name: tempo
  annotations:
    kubernetes.io/ingress.class: nginx-internal
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: GRPC    
spec:
  tls:
  - hosts:
    - otlp-grpc.tempo.internal.<hidden>.io
    secretName: le-prod-tempo
  rules:
  - host: otlp-grpc.tempo.internal.<hidden>.io
    http:
      paths:
      - path: /
        backend:
          serviceName: tempo-tempo-distributed-distributor
          servicePort: 4317

Seems to be working just fine for now. This uses the opensource nginx-ingress in the back + AWS Classic ELB (on private subnet shared between multiple clusters)

hi @BitProcessor , we are using same ingress setup but were wondering do you use 2 different ingress resources for read and write path ? I ask because reading from querier would not be over GRPC but over http, but the ingress has this annotation nginx.ingress.kubernetes.io/backend-protocol: GRPC

[Nello-Angelo]

any update ? how to setup ingress for tempo to send traces from more then from 1 cluster