Fetch all descriptors in one request and filter in code

[fredr]

The GCP pricing model for these APIs is to charge per request, so this patch would slightly decrease the number of requests in every scrape, and therefore reduce the cost a bit.

Preferably we could get all time series that we are interested in in one request, but unfortunately the API wont let us get time series from several descriptors in one request.

The default/max page size in not documented for this API, but it does return all of our descriptors in one page (over 4000 descriptors).

[weyert]

How can we progress this PR?

[SuperQ]

Hmm, I think things were done the current way in order to fetch large amounts of metrics faster, by executing them in parallel.

We needed to do this in order to reduce scrape intervals.

Have you tested how this impacts performance?

[fredr]

Hmm, I think things were done the current way in order to fetch large amounts of metrics faster, by executing them in parallel.

The metrics should still be fetched in parallel here: https://github.com/prometheus-community/stackdriver_exporter/pull/125/files#diff-b6f1dd37640aa1bdb3aff2c484e46fc67158d22598dca934306fa23185917da5R234

So I think this shouldn't affect the parallelism, if I remember correctly :thinking:

Have you tested how this impacts performance?

The employer I worked for have been running this since I wrote it, with no observed performance impact.

[SuperQ]

Sadly, I don't have a big stackdriver account to test this on anymore. Maybe @igorw can help with this?

[the-it]

@SuperQ recruited me to test this in a bit larger setup. The PR was deployed around 15:10 in the screenshot. The 2 hours before that is a bit off, due to difficulties during deployment.

[SuperQ]

Yes, that's a pretty big slowdown in colection. Perhaps we should instruement these phases of the collection so we know exactly how long it takes.

[fredr]

@the-it Just out of curiosity, do you know around what number of descriptors you are pulling?

I unfortunately don't manage a stackdriver setup any longer, so can't really investigate into this. IIRC this fix reduced the price a bit, but the bulk of the requests comes from fetching the metrics, as that is one request per metric, so in the grand scheme of things it might not be worth it if you see a performance penalty.

[WojciechKuk]

@

Hmm, I think things were done the current way in order to fetch large amounts of metrics faster, by executing them in parallel.

using stackdriver_exporter on 50+ projects I get throttled by GCP (max 6000rpm allowed per service account) just after ONE request to /metrics endpoint.

@SuperQ Is it possible to add some --parallel=X parameter?

[SuperQ]

Maybe this is enough: https://github.com/prometheus-community/stackdriver_exporter/pull/218

[fredr]

Maybe this is enough: #218

Nice! I'll close this one in favor of that one!