pyroscope POD to POD communication is not happening with Istio proxy enabled in AKS

We are using grafana/pyroscope version v1.2.0. We are running Pyroscope in microservice mode in AKS cluster. We are having istio proxy enabled in our AKS cluster. When istio is enabled, Pod to Pod communication is not happening between the components using their Pod IP address. We are getting 502 bad gateway error when istio proxy is enabled

Steps to reproduce the behavior:

Start Pyroscope Pod in AKS Push profiles from sample application to Pyroscope's distributor endpoint Query in grafana using Pyroscope's query-frontend endpoint

Expected behavior Pyroscope components - distributor, ingester, store gateway, querier, query scheduler, compactor and query frontend should communicate to each other using their Pod IP address.

Environment Infrastructure: Kubernetes (AKS) Deployment tool: Helm

Dec 11 '23 21:12 Raghav65

@simonswine Can you please help here. We are blocked on this issue for a long time

Dec 20 '23 17:12 Raghav65

Thanks for reporting @Raghav65 we will try to fit this into our roadmap but as of right now do not have a timeline on when we will be able to implement a solution.

We will update here as we make progress

Dec 20 '23 21:12 Rperry2174

Hello @Rperry2174 Thanks for the response. Any luck on the timelines on when this solution can be implemented? This could highly help many who are interested to use Pyroscope in their technology stack

Jan 03 '24 18:01 Raghav65

I won't be able to provide a time line for this, but most likely the solution will be looking somewhat similar to how it is documented for loki:

https://grafana.com/docs/loki/latest/setup/install/istio/

Jan 08 '24 17:01 simonswine