icingaweb2-module-director icon indicating copy to clipboard operation
icingaweb2-module-director copied to clipboard

Published 20 hours ago • verified publisher icon Icinga

Group import using LDAP deletes groups when limit of 1000 is reached

Open laugmanuel opened this issue 2 years ago • 3 comments

We use an LDAP resource together with the director LDAP import feature to import and use many usergroups. Yesterday, I noticed that some groups were deleted in Icinga director after the import source was executed. The same run of the import source also created a new group.

When downloading the JSON preview of the import source (Director -> Import source -> Preview), I can see exactly 1000 groups being shown:

❯ jq '. | keys | length' Downloads/director-importsource-27_20220727140303.json
1000

The deleted groups also seem to be alphabetically at the end of all groups matching the group filter.

Expected Behavior

The import source should import all groups matching the group filter and not just the first 1000.

Current Behavior

Only the first 1000 groups are returned. Groups not part of that list will be deleted in Icinga.

Possible Solution

Pagination was introduced in (or at least it looks like it): https://github.com/Icinga/icingaweb2-module-director/issues/2019

Steps to Reproduce (for bugs)

  • Create a import source matching less than 1000 groups
  • Import all groups into Icinga
  • Change the group filter to match more than 1000 groups
  • Run the import again
  • only the first 1000 groups (sorted alphabetically) will remain or get created; all other groups will be deleted

Your Environment

  • Director version (System - About): 1.9.1 and master
  • Icinga Web 2 version and modules (System - About): 2.9.6
  • Icinga 2 version (icinga2 --version): r2.13.3-1
  • Operating System and version: RHEL 7.9
  • Webserver, PHP versions: httpd 2.4.6-97 and PHP 7.3.33

laugmanuel avatar Jul 27 '22 12:07 laugmanuel

As a short-term relieve, I've added a second import source and split the import into two distinct groups. Also, I've extended the LDAP filter to only import groups containing members - empty groups are not required for our usecase.

laugmanuel avatar Jul 27 '22 12:07 laugmanuel

The following modification of "/usr/share/icingaweb2/modules/director/library/Director/Import/ImportSourceLdap.php" solves this behavior:

- ->setUsePagedResults()
+ ->setUsePagedResults(false)

Would be nice to have it upstream.

mwaldmueller avatar Aug 11 '22 07:08 mwaldmueller

Tried the workaround but did not help. Digging into the code it looks correct for me, when the query should use paging and the server has the capability it should use paging when fetching all results and return not only 1000 items.

Enabled debug logging and verified the settings to get the capabilities which include 1.2.840.113556.1.4.319 which is used at https://github.com/Icinga/icingaweb2/blob/master/library/Icinga/Protocol/Ldap/LdapCapabilities.php#L141-L149

2022-08-16T09:54:05+02:00 - DEBUG - Connect using LDAPS
2022-08-16T09:54:06+02:00 - DEBUG - LDAP query result does not provide the requested field "vendorName"
2022-08-16T09:54:06+02:00 - DEBUG - LDAP query result does not provide the requested field "vendorVersion"
2022-08-16T09:54:06+02:00 - DEBUG - LDAP query result does not provide the requested field "objectVersion"
2022-08-16T09:54:06+02:00 - DEBUG - LDAP query result does not provide the requested field "+"
2022-08-16T09:54:06+02:00 - DEBUG - Capability query discovered the following attributes:
2022-08-16T09:54:06+02:00 - DEBUG -  supportedExtension = ["1.3.6.1.4.1.1466.20037","1.3.6.1.4.1.1466.101.119.1","1.2.840.113556.1.4.1781","1.3.6.1.4.1.4203.1.11.3","1.2.840.113556.1.4.2212"]
2022-08-16T09:54:06+02:00 - DEBUG -  supportedControl = ["1.2.840.113556.1.4.319","1.2.840.113556.1.4.801","1.2.840.113556.1.4.473","1.2.840.113556.1.4.528","1.2.840.113556.1.4.417","1.2.840.113556.1.4.619","1.2.840.113556.1.4.841","1.2.840.113556.1.4.529","1.2.840.113556.1.4.805","1.2.840.113556.1.4.521","1.2.840.113556.1.4.970","1.2.840.113556.1.4.1338","1.2.840.113556.1.4.474","1.2.840.113556.1.4.1339","1.2.840.113556.1.4.1340","1.2.840.113556.1.4.1413","2.16.840.1.113730.3.4.9","2.16.840.1.113730.3.4.10","1.2.840.113556.1.4.1504","1.2.840.113556.1.4.1852","1.2.840.113556.1.4.802","1.2.840.113556.1.4.1907","1.2.840.113556.1.4.1948","1.2.840.113556.1.4.1974","1.2.840.113556.1.4.1341","1.2.840.113556.1.4.2026","1.2.840.113556.1.4.2064","1.2.840.113556.1.4.2065","1.2.840.113556.1.4.2066","1.2.840.113556.1.4.2090","1.2.840.113556.1.4.2205","1.2.840.113556.1.4.2204","1.2.840.113556.1.4.2206","1.2.840.113556.1.4.2211","1.2.840.113556.1.4.2239","1.2.840.113556.1.4.2255","1.2.840.113556.1.4.2256","1.2.840.113556.1.4.2309","1.2.840.113556.1.4.2330","1.2.840.113556.1.4.2354"]
2022-08-16T09:54:06+02:00 - DEBUG -  supportedCapabilities = ["1.2.840.113556.1.4.800","1.2.840.113556.1.4.1670","1.2.840.113556.1.4.1791","1.2.840.113556.1.4.1935","1.2.840.113556.1.4.2080","1.2.840.113556.1.4.2237"]
2022-08-16T09:54:06+02:00 - DEBUG -  supportedLDAPVersion = ["3","2"]
...

Perhaps this helps to find an error.

dgoetz avatar Aug 16 '22 08:08 dgoetz

The suggested workaround depends on your server configuration, might work for OpenLDAP, doesn't for AD. LDAP pagination has been broken for PHP 7.3+ and will be fixed with the pull request linked by @lippserd. We successfully tested the fix on RHEL and Debian with different LDAP server implementations (OpenLDAP, IBM).

Closing the issue here, please follow #4901 for the related Icinga Web fix or manually apply the patch in the meantime.

Thomas-Gelf avatar Sep 01 '22 10:09 Thomas-Gelf