[FEAT]: Support filter parameter on github_organization_team_sync_groups data source

[janslow]

Describe the need

Currently, the github_organization_team_sync_groups data source fetches all groups (in fixed page sizes of 100). This causes performance issues if a large number of groups are synced into GitHub (we have about 3000 groups being pushed to GitHub, so the refresh of github_organization_team_sync_groups takes 20+ seconds).

As a feature request, please expose the q query parameter, to allow the data source to be scoped to a subset of all groups:

Filters the results to return only those that begin with the value specified by this parameter. For example, a value of ab will return results that begin with "ab".

For example,

data "github_organization_team_sync_groups" "filtered" {
  q = "myprefix_"
}

SDK Version

No response

API Version

REST API 2022-11-28 (Enterprise Cloud)

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[cardoe]

Could this be related to #560?

[janslow]

Could this be related to #560?

Not as far as I can see.

The #560 issue seems to be related to a server error from the Github API, so probably can't be solved by a change in the Terraform provide

This issue is a request to expose an existing feature of the Github API via the provider

There doesn't seem to be any overlapping symptoms between the two

[github-actions[bot]]

👋 Hey Friends, this issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Please add the Status: Pinned label if you feel that this issue needs to remain open/active. Thank you for your contributions and help in keeping things tidy!

[laughedelic]

This is still very much relevant. Can we reopen it? The data source in the current state is not usable when the IdP has thousands of groups, it should be possible to filter the groups by id or name.

[laughedelic]

This is a limitation of the go-github library, so I opened a request upstream: https://github.com/google/go-github/issues/3196 / https://github.com/google/go-github/pull/3197

[laughedelic]

Since fixing this properly takes a long time, here is my workaround using the github_rest_api data source:

data "github_rest_api" "team_sync_groups" {
  endpoint = "orgs/ORG/team-sync/groups?q=your%20prefix"
}

locals {
  # index IdP groups by group name:
  idp_sync_groups_lookup = { for group in jsondecode(data.github_rest_api.team_sync_groups.body).groups : group.group_name => group }
}

Then you can do local.idp_sync_groups_lookup["your prefix group name"] to get the group data.

Note that if you have spaces or any other non-URL symbols in the prefix, you need to url-encode them.

---

Update: this might be only a partial workaround, because this data source doesn't seem to do auto-pagination. The endpoint returns 30 results per page by default, so if you expect more, you can add &per_page=100 at the end.

[fatbasstard]

@laughedelic Since your upstream PR's are merged a long time ago already, any chance we can get this in here. Not sure who to poke exactly so going with you ;)

[laughedelic]

Hi @fatbasstard. Initially, I waited for the upstream library version update. Now it looks like the currently used version (66) includes those changes. So it shouldn't be a problem. I don't have time to contribute this change here though