Partial group membership list retrieved from gitlab

[sbasgall]

Looking at the debug web logs when a user logs in via gitlab authorization only a partial list of groups is returned.

I see these gitlab api calls in the logs https://my.gitlab.server/api/v4/groups?order_by=name&owned=false&page=2&per_page=20&sort=asc&statistics=false&with_custom_attributes=false

https://my.gitlab.server/api/v4/groups?order_by=name&owned=false&page=1&per_page=20&sort=asc&statistics=false&with_custom_attributes=false

https://my.gitlab.server/api/v4/groups?order_by=name&owned=false&page=13&per_page=20&sort=asc&statistics=false&with_custom_attributes=false

It seems like many pages are skipped. The list that is returned is much smaller than expected even for 3 pages worth of group retrieval. It is exactly 20 groups - the first 20 groups in gitlab alphabetically.

I would expect all the pages to be iterated over not just the first, second and last. Is there a configuration setting that could help with this?

What additional information would you like?

[xuwang]

We hit the same bug, groups only in first 20 will be synced. All other gitlab groups will be missed. sonarqube v7.1.0.11001 sonar-auth-gitlab-plugin v1.3.2

[daniel-ng]

Having the same issue. I closed mine and will piggy back off of this one. I can't properly manage my users because the plugin only grabs the first 20 groups.

[daniel-ng]

This is new to me - I'm doing a little bit of digging in my spare time. Trying to trace this down. Looks like maybe it's because the GitLab API only returns 20 groups when making the request to /groups? I am trying to figure out where in java-gitlab-api it actually steps through the groups, but hopefully this can provide a little bit of a lead? Or perhaps someone with more java know-how can tell me if this is doing what it's supposed to be doing?

Stream.of(api.getGitLabAPIGroups().getMyGroups()).map(Paged::getResults).flatMap(Collection::stream).map(com.talanlabs.gitlab.api.v4.models.GitlabGroup::getName)
                        .collect(Collectors.toSet())

[daniel-ng]

Unfortunately, I don't have the cycles to look into this more any time soon.

I'm curious - how are you guys using group permissions? Do you create and assign projects to a specific org each time? Or is it specified at the time that you run the test?

[daniel-ng]

Any updates on this? @gabrie-allaigre any suggestions or ideas for fixes/workarounds?

[bromine0x23]

It does read only the first page because Paged::getResults just return items of current page but not iterate on all pages. It should use Paged::nextPage to read all pages or use Paged::iterator to get a iterator.

Unfortunately the current used GitLab API library com.talanlabs:java-gitlab-api:1.4.1 has a bug on Paged::iterator which hasNext use a wrong condition, so the final implement maybe like that:

Stream.iterate(
	api.getGitLabAPIGroups().getMyGroups(),
	Objects::nonNull,
	p -> {
		try {
			return p.nextPage();
		} catch (IOException exception) {
			throw new RuntimeException(exception);
		}
	}
)
.map(Paged::getResults)
.flatMap(Collection::stream)
.map(com.talanlabs.gitlab.api.v4.models.GitlabGroup::getName)
.collect(Collectors.toSet());