invalid_argument: protocol error: incomplete envelope: stream error: stream ID xxxx

[Raghav65]

We are using grafana/pyroscope version v1.2.0. We are running Pyroscope in microservice mode. All pods (distributor, ingester, store gateway, querier, compactor, query scheduler and query frontend) are running.

Please find the pyroscope config in the attachment for all components. code : pyroscope_config.txt

How to performance test write path and read path in Pyroscope for any sample application?

For example we are pushing profiles using simple java app, we can see 1 Bil samples per second for all duration for process_cpu profile type in grafana in read path.

https://github.com/grafana/pyroscope/tree/main/examples/java/simple

Few test suites we tested below for 5 minutes interval:

1. 3 functions-Different Application Name-Same Tenant- 100 replicas - 27.53 Bil samples/sec
2. 100 Functions-Same Application Name-Same Tenant-100 replicas - 14.23 Bil samples/sec
3. 100 Functions-Different Application Name-Same Tenant-100 replicas - 26.6 Bil samples/sec

Increasing the replicas or changing the application name doesn't scale the load.

How to control number of profile samples to be pushed in write path ?

Also we are getting below error running read path testing in jmeter. unknown: invalid_argument: protocol error: incomplete envelope: stream error: stream ID 4001; CANCEL unknown: invalid_argument: protocol error: incomplete envelope: i/o timeout

Appreciate if we get assistance as soon as possible

[cyriltovena]

For load testing I wouldn't suggest to use the java/simple. We don't have the right tool for this so I would encourage to send some dev data from your workload instead.

unknown: invalid_argument: protocol error: incomplete envelope: stream error: stream ID 4001; CANCEL
unknown: invalid_argument: protocol error: incomplete envelope: i/o timeout

Without knowing the type of query or logs it's hard to tell.