Pyroscope write to remote endpoint [AWS ALB] returns 405 Method Not Allowed

[codevally]

trafficstars

Describe the bug

We are using Grafana alloy to push. profiles to a remote Pyroscope endpoint (backed by AWS ALB), but alloy throwing this error. ts=2024-06-27T10:04:34.401994086Z level=warn msg="failed to push to endpoint" component_path=/ component_id=pyroscope.write.endpoint endpoint=https://profiling.remote.url.com/ingest err="unknown: 405 Method Not Allowed"

To Reproduce

Steps to reproduce the behavior: Remote pyroscope behind AWS ALB and Grafana alloy push profiles vis pyroscope.write

1. Start Pyroscope (SHA or version) 1.6.0
2. Perform Operations(Read/Write/Others) Write

Expected behavior

Remove Pyroscope distributor component should receive the profile data without an erro/warn.

Environment

• Infrastructure: [e.g., Kubernetes, bare-metal, laptop] Kubernetes
• Deployment tool: [e.g., helm, jsonnet] Helm

Additional Context

[simonswine]

If you still experiencing those errors, it would be interesting how you configured the ingest to your cluster. If you can share some yamls for that and maybe the values.yaml of your helm deployment.

[venkivijay]

This issue might be due to the AWS ALB rule prioritization. Pyroscope exposes multiple endpoints through its ingress, such as:

/ -> pyroscope-query-frontend:4040
/querier.v1.QuerierService/ -> pyroscope-query-frontend:4040
/render -> pyroscope-query-frontend:4040
/render-diff -> pyroscope-query-frontend:4040
/push.v1.PusherService/ -> pyroscope-distributor:4040
/ingest -> pyroscope-distributor:4040
/settings.v1.SettingsService/ -> pyroscope-tenant-settings:4040
/adhocprofiles.v1.AdHocProfileService/ -> pyroscope-ad-hoc-profiles:4040

When the AWS Load Balancer Controller creates rules for these ingress endpoints, the priority of the rules is not guaranteed. As a result, the / route (frontend) may take precedence over the /ingest or /push.v1.PusherService/ endpoints. So, when profiling data is sent, it could be routed to the frontend pod instead of the distributor, causing the 405 Method Not Allowed error.

This issue can be resolved by implementing an NGINX gateway, similar to the approach used for services like Tempo, Loki, and Mimir. This would centralize the routing through a single entry point, allowing NGINX to handle the internal routing to the appropriate Pyroscope components. This solution would ensure proper routing and eliminate the rule priority conflict. There is already an open issue for this. https://github.com/grafana/pyroscope/issues/2082