Does ldapsdk provide method which can get fresh(period time) data from Action Directory?

[TimmyWangDouDou]

I want to get the data of the last one day ,two days or some period time from AD,which class or method in ldapsdk could realize that function?

[dirmgr]

There are a few of ways to detect changes in Active Directory. The best options are described at https://support.microsoft.com/en-us/help/891995/how-to-poll-for-object-attribute-changes-in-active-directory-on-window and include:

• You can use the DirSync control, which the LDAP SDK supports in the com.unboundid.ldap.sdk.experimental.ActiveDirectoryDirSyncControl class. This is probably the preferred option.

• You can issue search requests with filters that target the uSNChanged attribute. Each AD instance maintains an integer value that increments each time an entry is added or updated, and the value of that counter gets stored in the uSNChanged attribute in the target entry.

[nddipiazza]

@dirmgr I have this same question. How can I send a timestamp or initial cookie into the ActiveDirectoryDirSyncControl?

(let me know if you want this as a new issue)

My use case is the following:

1. First fetch all users and groups from AD.

2. Now that I have all the users and groups, I want to store a token or timestamp that represents when the user/group dump was done.

3. Every 10 minutes, fetch all the new/updated/deleted users from ldap since the last cookie/timestamp.

I am using DirSync to fetch updates using example here: https://docs.ldap.com/ldap-sdk/docs/javadoc/com/unboundid/ldap/sdk/experimental/ActiveDirectoryDirSyncControl.html

But when I do that it fetches me all deleted records since the start of time! Which is a lot in my case.

Can I somehow do a full fetch, then send it into the dirsync?

Full fetch example:

@Override
  public Optional<byte[]> getUsers(
      String baseDN,
      int pageSize,
      byte[] cookie,
      Consumer<String> userConsumer
  ) throws Exception {
    ASN1OctetString resumeCookie = cookie.length == 0 ? null : new ASN1OctetString(cookie);
    SearchRequest searchRequest = new SearchRequest(
        baseDN,
        SearchScope.SUB,
        "(objectClass=user)",
        LdapUtil.USER_PRINCIPAL_NAME, LdapUtil.SAM_ACCOUNT_NAME, LdapUtil.MEMBER_OF
    );
    searchRequest.setControls(new SimplePagedResultsControl(pageSize, resumeCookie));

    SearchResult searchResult = search(searchRequest);

    for (SearchResultEntry searchResultEntry : searchResult.getSearchEntries()) {
        userConsumer.accept(
            searchResultEntry.getDN()
        ));
    }
    SimplePagedResultsControl responseControl = SimplePagedResultsControl.get(searchResult);

    if (responseControl != null && responseControl.moreResultsToReturn()) {
      return Optional.of(responseControl.getCookie().getValue());
    } else {
      return Optional.empty();
    }
}

Is there a way to take a cookie or timestamp from when we fired off this full fetch and feed the "cookie" (or timestamp) to DirSync so i can get changes made since the full fetch?

[dirmgr]

I'm really not at all familiar with Active Directory. However, my understanding from looking at the documentation that Microsoft provides, and from how other controls work, is that you have to provide the control without a cookie the first time that you use it, and that has the potential to return a lot of results. However, that should give you a cookie value that you can provide back to the server on subsequent uses to just get what has changed since you got that cookie. Microsoft does not document the format for the cookie and explicitly states that the client should not attempt to interpret it or rely on its format.

I honestly don't know whether the cookie is tied to the specific filter that you used (I don't see anything in the documentation that says that is the case), but if you don't want any historical results, then you could try performing the initial search with a filter that won't match anything, just so you can get a cookie, and then use that cookie to resume the search with the filter that you really want.

[nddipiazza]

@dirmgr I thought the same thing. Great minds think alike! But the problem is that the cookie that it returns is the next page at 1000 entries a pop. So you have to run it all the way through until empty. Kind of weird api from microsoft. you'd figure any incremental api would allow some way to send a timestamp.

[vdailly]

Normally when providing the DirSync control with a null cookie, you get the whole results at first. Then on subsequent searchs, you provide the previously stored cookie and the directory server provide only what was updated. But ... depending of a lot of factors (attributes requested, the filter ...) this is not always as easy. This is not especially due to the control but how it works internally.

As examples (this is a gift): Example 1: if you request (objectclass=user)(objectcategory=person), then you wouldn't get back the Deleted Objects ... because deleted objects lose their objectcategory attribute. This is clearly stated in the documentation MS-ADTS. So you cannot easily split computer and user accounts ... :)

Example 2: when you request member attribute, a group object may be splitted in multiple search results, each with the DN, the objectguid, and some fancy member;range=x y attribute name. And for each member you don't know if the member was added or deleted. It's up to you to guess depending of your context (sync to DB, to another LDAP ... and so on).

Example 3: if you want to include objectclass attribute in results, this would be the case when you request the directory with this control the first time. But on subsequent this attribute would never appear because it does never change. For this Microsoft provide a DirSyncExtented control, provided the directory server operating system support it. And you have to use fancy syntax to force the domain controller to return this attribute. This is documented here

The DirSync is highly coupled with Active Directory replication mechanisms. Globally, the cookie store the highest USN number of the current DC requested, as well as all known USN of all others Active Directory server in the domain/forest (provided network is full meshed ...).

The detailed structure of the cookie is not well described. But globally the attribute replUpToDateVector on the domain NC is the information stored in the cookie. This is a "replication" state. So even if the next time you provide the cookie to another server, this new server will answer with the modifications greater than its own USN that was stored in the initial server. This is very powerful and you even don't need a connection pool. You just have to setup a LDAP connection to the DNS name of your domain. So you don't care to request a specific DC.

The detailed documentation about the control is provided here. The controlValue is the BER encoding of the following ASN.1 structure DirSyncResponseValue ::= SEQUENCE { MoreResults INTEGER unused INTEGER CookieServer OCTET STRING }

With the .NET API there is a class used as response control: dirsyncresponsecontrol

The steps are as follow:

• create the search request
• create the dirsyncrequestcontrol
• add the dirsyncrequestcontrol to the search request
• now loop (even indefinitly ...)
  • send the search request, waiting a search response.
  • process the results ...
  • get the returned controls if the "MoreResults" field of the returned control is 0 then the value is the last value and should be stored for the next time. If this is 1, then you just update the dirsyncrequestcontrol cookie value with the "Cookie Server" field value of the returned control. This is globally the same behaviour than for pagination. Just ensure to exit the loop :)

This works well with the .NET API System.DirectoryServices.Protocols.

The cookie structure is described here. As stated in this GetResponseDirSyncControlValue the cookie value is computed from internal call to the Directory Replication Service Remote Protocol.

Maybe some already provided information:

• https://docs.microsoft.com/en-us/windows/win32/ad/polling-for-changes-using-the-dirsync-control?redirectedfrom=MSDN