[tempo] helm chart is not ready for monolithic scaling

[dastrobu]

I am currently attempting to run Grafana Tempo in monolithic mode with scaling, following the instructions provided in the Scaling monolithic mode documentation.

However, it seems that the helm chart is not well-prepared for this scenario or lacks sufficient documentation.

According to the Scaling monolithic mode guide, the -target parameter needs to be set to scalable-single-binary. This can be achieved by adding the following to the values in the YAML file:

    extraArgs:
      target: scalable-single-binary

Unfortunately, the pods fail to start up and display the following error message:

level=error ts=2024-04-22T08:32:45.730366914Z caller=main.go:121 msg="error running Tempo" err="failed to init module services: error initialising module: querier: frontend worker address not specified"

Additionally, it appears that a headless service is required to configure the memberlist.join_members property, as mentioned in the memberlist configuration documentation. However, the headless service is either missing from the helm chart or there is a misconfiguration, as indicated in issue #3042.

It would be greatly appreciated if the helm chart could support scalable-single-binary or if the readme could include a statement clarifying that only singleton deployments are currently supported.

[algo7]

Any news on this?

[lieberlois]

+1 Also interested in this

[qinkangdeid]

I'm also interested in this and would like to know how to support scaling in monolithic mode.

[jpicara]

Has anyone found the solution? I am trying also to deploy over helm chart the scaling monolith mode in order to have different tempo replicas being written at the same time and got the same error message adding the -target=scalable-single-binary. err="failed to init module services: error initialising module: querier: frontend worker address not specified"

[dastrobu]

I ended up switching to microservices mode.

[moshen-nemodata]

Hi, Any plans to fix this, would be great if we could have something simple scalable Helm chart to Loki simple scalable Helm chart

[marijakljestan]

Any updates on this?

[Mesverrum]

My understanding is Grafana doesn't actually use the monolithic modes of any of their back ends in any serious way, so they don't spend much time focused on it and the expectation is the community will come up with their own solutions. Or adopt the microservices mode that the Grafana engineers actually use in their day jobs. So in short, you will probably be waiting a long time if you want to passively wait for someone at Grafana to work on this. Or you'll want to work it out for yourself and share it back.