feat(grafana): Use pagination for Grafana Dashboard search

[robbat2]

Hey, I just made a Pull Request!

Grafana's Search API endpoint applies a default of limit of 1000 results - cutting off any dashboards past that point. Apply a trivial pagination wrapper to fetch up to 1M dashboards.

Reference: https://github.com/backstage/community-plugins/issues/3735#issuecomment-2832849034

:heavy_check_mark: Checklist

• [X] A changeset describing the change and affected packages. (more info)
• [ ] Added or updated documentation
• [ ] Tests for new functionality and regression tests for bug fixes
• [ ] Screenshots attached (for UI changes)
• [X] All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie[bot]]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage-community/plugin-grafana	workspaces/grafana/plugins/grafana	minor	v0.6.0

[robbat2]

@vinzscam ping

[robbat2]

thank you for the contribution. Is it possible to fetch additional dashboards once the user scrolls to the bottom of the page instead of fetching everything in one shot?

That statement would only make sense if they had a huge number of dashboards (1000+) that matched the dashboard selector, and the Backstage entity page would likely be unusable in that case. I think it's extremely unlikely to occur.

The most common situation is:

Searching for tag "X".

The dashboard API has this in total: Dashboard 0: tag ... Dashboard 1: tag X Dashboard 999: tag ... Dashboard 1001: tag X

The query, with default limit 1000 returns dashboard 0-999 inclusive, which are then filtered client-side to match tag X, leaving only dashboard 1.

Dashboard 1001 is not listed on the Backstage entity page.

In the github issue https://github.com/backstage/community-plugins/issues/3735 - I proposed a further improvement, but it should built on TOP of this PR, rather than be conflated:

Specifically, parse the dashboard selector, and extract the tags to improve the queries to Grafana.

E.g, from the example on https://github.com/backstage/community-plugins/blob/master/workspaces/grafana/plugins/grafana/docs/dashboards-on-component-page.md grafana/dashboard-selector: "(tags @> 'my-service' || tags @> 'my-service-slo') && tags @> 'generated'"

Perform 2 paginated queries to Grafana's Dashboard API:

1. tag=my-service&tag=generated
2. tag=my-service-slo&tag=generated

Combine the results per the selector, removing duplicates

[robbat2]

@vinzscam ping

[robbat2]

@vinzscam ping. What is needed to unblock this? As I noted in the previous answer, this PR is needed when the Grafana deployment has many dashboards, regardless of how few of them are related to a Backstage entity.

In one case raised by a co-worker today, this makes the difference between incorrectly showing zero dashboards, vs the two dashboards for that entity.

[robbat2]

@vinzscam ping

[vinzscam]

thank you for the contribution. Is it possible to fetch additional dashboards once the user scrolls to the bottom of the page instead of fetching everything in one shot?

That statement would only make sense if they had a huge number of dashboards (1000+) that matched the dashboard selector, and the Backstage entity page would likely be unusable in that case. I think it's extremely unlikely to occur.

The most common situation is:

Searching for tag "X".

The dashboard API has this in total: Dashboard 0: tag ... Dashboard 1: tag X Dashboard 999: tag ... Dashboard 1001: tag X

The query, with default limit 1000 returns dashboard 0-999 inclusive, which are then filtered client-side to match tag X, leaving only dashboard 1.

Dashboard 1001 is not listed on the Backstage entity page.

In the github issue #3735 - I proposed a further improvement, but it should built on TOP of this PR, rather than be conflated:

Specifically, parse the dashboard selector, and extract the tags to improve the queries to Grafana.

E.g, from the example on https://github.com/backstage/community-plugins/blob/master/workspaces/grafana/plugins/grafana/docs/dashboards-on-component-page.md grafana/dashboard-selector: "(tags @> 'my-service' || tags @> 'my-service-slo') && tags @> 'generated'"

Perform 2 paginated queries to Grafana's Dashboard API:

1. tag=my-service&tag=generated
2. tag=my-service-slo&tag=generated

Combine the results per the selector, removing duplicates

~then what's the point of fetching 1M dashboards? Can't we just fetch a single page and simplify the code even more?~

ok I understand the issue, @robbat2. But this looks like an issue on grafana side. Can you share the version of grafana you are using?

[robbat2]

Grafana 10.4.19 Enterprise now; previously produced on older versions of Grafana.

It's not a Grafana bug - it's a bug in the plugin because the client fetches only the first page of dashboards from Grafana, without pagination and without any search queries, and then does a client-side filter of the returned dashboards.

The client does not fetch any further pages of dashboards from Grafana, so if those pages contained matching dashboards for the client-side filter, the results are not displayed.

In the original issue, I suggested a deeper improvement: extracting the tags from the annotation and using them to perform better client fetches of dashboards; but that really should be a followup PR to this.

[vinzscam]

Grafana 10.4.19 Enterprise now; previously produced on older versions of Grafana.

It's not a Grafana bug - it's a bug in the plugin because the client fetches only the first page of dashboards from Grafana, without pagination and without any search queries, and then does a client-side filter of the returned dashboards.

The client does not fetch any further pages of dashboards from Grafana, so if those pages contained matching dashboards for the client-side filter, the results are not displayed.

In the original issue, I suggested a deeper improvement: extracting the tags from the annotation and using them to perform better client fetches of dashboards; but that really should be a followup PR to this.

Thanks for clarifying, @robbat2. 🙏

I think a good way to move forward would be to implement some sort of pagination on the frontend, so that the frontend requests n dashboards and requests more as needed. I'm a bit worried about fetching all the dashboards in one shot, as this could be an issue for big companies.

[robbat2]

The dataset of the list of dashboards is fully indexed & heavily cached in Grafana, because it powers Grafana's main dropdown, as well being lazy-fetched when users hit the Grafana /dashboard page in the web interface and often used on the front page. We're only recently exceeding 1500 dashboards at $JOB, so that would be 2 requests for 1000 dashboards each - doubling the number of requests that this plugin makes today.

It's hard for the frontend to know: has it seen all of the dashboards that would be output after the client-side filtering has run.

Consider if the annotation contains a typo of the tag, and we had logic that said "load pages from Grafana until you have at least 1 dashboard or reach the end of pages" - it would always load all pages from Grafana.

I will commit to a future improvement that extracts the tags and queries based on them, but I want to get this minimal fix out first.

[vinzscam]

The dataset of the list of dashboards is fully indexed & heavily cached in Grafana, because it powers Grafana's main dropdown, as well being lazy-fetched when users hit the Grafana /dashboard page in the web interface and often used on the front page. We're only recently exceeding 1500 dashboards at $JOB, so that would be 2 requests for 1000 dashboards each - doubling the number of requests that this plugin makes today.

It's hard for the frontend to know: has it seen all of the dashboards that would be output after the client-side filtering has run.

Consider if the annotation contains a typo of the tag, and we had logic that said "load pages from Grafana until you have at least 1 dashboard or reach the end of pages" - it would always load all pages from Grafana.

I will commit to a future improvement that extracts the tags and queries based on them, but I want to get this minimal fix out first.

ok I think a good way to move forward is to deprecate the existing annotation and create another one which performs server side filtering using the query params available in Grafana.

Your solution works for you but it might not work for adopters at a bigger scale

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[kmcrawford]

I'm having the same issue..

[robbat2]

@vinzscam as a way forward before that more invasive change of changing the annotation - would you accept a maxpages configuration option, that defaults to "1"; that would keep the existing behavior the same (a single request, return 1000 entries).

Then integrators could opt to raise that value in their own environments in line with their own scale needs [I'd set it to "3" right now, to cover current dashboards and give a little growth room].

[vinzscam]

@vinzscam as a way forward before that more invasive change of changing the annotation - would you accept a maxpages configuration option, that defaults to "1"; that would keep the existing behavior the same (a single request, return 1000 entries).

Then integrators could opt to raise that value in their own environments in line with their own scale needs [I'd set it to "3" right now, to cover current dashboards and give a little growth room].

that's fine. Actually you could expose just a grafanaDashboardSearchLimit config, skipping the maxpages option for now, as it's possible to specify a max limit of 5000 according to the official docs.

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!