Support IAsyncEnumerable out of the box

[twsouthwick]

Is your feature request related to a problem? Please describe. This is not so much a problem, but difficult to implement as there is no abstracted concept of page (see https://github.com/microsoftgraph/msgraph-sdk-dotnet/issues/511). Every request has its own GetAsync(CancellationToken) and each page has its own NextPageRequest.

Describe the solution you'd like I'd love to be able to do something like the following:

g.Groups["some id"]
  .Members
  .Request()
  .ToAsyncEnumerable(token);

This allows for processing in memory of what needs to be done, but the paging occurs without any intervention on command if needed.

Describe alternatives you've considered I've implemented this manually for some of the collections I care about. For example:

public static IAsyncEnumerable<string> GetUserIds(this IGraphServiceClient g, CancellationToken token)
{
    return g.Groups["some id"]
          .Members
          .Request()
          .ToAsyncEnumerable(token)
          .Select(d => d.Id);
}

private static async IAsyncEnumerable<DirectoryObject> ToAsyncEnumerable(this IGroupMembersCollectionWithReferencesRequest request, [EnumeratorCancellation] CancellationToken token)
{
    if (request is null)
    {
        throw new ArgumentNullException(nameof(request));
    }

    do
    {
        var page = await request.GetAsync(token).ConfigureAwait(false);

        foreach (var item in page)
        {
            yield return item;
        }

        request = page.NextPageRequest;
    } while (request != null);
}

AB#7248

[pschaeflein]

@twsouthwick - Would you consider adding this to the Graph Community library? Or would you object if I added it?

[twsouthwick]

This seems like something that would make sense in the core library as it could be autogenerated along with the other APIs here. If there is not interest to add it here, I could help add in the Graph Community Library (I didn't know that was available!)

[pschaeflein]

The community library is intended to support temporary code before it gets added to the Graph service or native SDK. We monitor the SDK and remove code that gets added.

Thanks!

[twsouthwick]

Ah - that's good to know! That sounds like a good place for it. I'm happy to do it, but it may be a few days before I can get to it. No object if you want to do it before then!

[pavram]

Sorry for barfing this up, but this is the wonderful hack I've been thinking about making less dangerous.

I'm going to go look for the post where this was raised in the community library now instead of using this.

Particularly wonderful things to take note of.

T can't implement anything of value thanks to the interfaces building the return types uniquely!

Example usage:

await foreach (Event calendarEvent in events.ReadEvents<IUserEventsCollectionPage,Event>()) {
// do a thing.
}

take note; that you have to manually specify a type for the collection (Y) because it can't be determined at compile time.

   public static async IAsyncEnumerable<Y> ReadThingsHorribly<T,Y>(this T startingPage,
                               [EnumeratorCancellation] CancellationToken cancellationToken = default)
            where T : IList<Y>
            {

            T page = startingPage;

            while (page != null) {
                foreach (var item in page) {
                    yield return item;
                }
                var nextPageRequest = page.GetType().GetProperty("NextPageRequest").GetValue(page);
                var getAsyncMethod = nextPageRequest.GetType().GetMethod("GetAsync", new Type[] { cancellationToken.GetType() });
                Task task = (Task)getAsyncMethod.Invoke(nextPageRequest, new object[] { cancellationToken });
                await task;

                var resultTypeProperty = task.GetType().GetProperty("Result");
                var result = resultTypeProperty.GetValue(task);
                page = (T)result;
            }
        } 

[LeonarddeR]

I wrote my own before I discovered the suggested approach from @pavram. Sharing for reference:

public static async IAsyncEnumerable<TEntity> AsAsyncEnumerable<TEntity>(this ICollectionPage<TEntity> collectionPage, [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        ICollectionPage<TEntity>? currentPage = collectionPage;
        String? nextLink = null;
        do
        {
            foreach (TEntity? item in currentPage)
            {
                cancellationToken.ThrowIfCancellationRequested();
                yield return item;
            }
            dynamic page = currentPage;
            if (page.NextPageRequest == null || nextLink == page.NextPageRequest.GetHttpRequestMessage().RequestUri.AbsoluteUri)
            {
                currentPage = null;
                yield break;
            }
            nextLink = page.NextPageRequest.GetHttpRequestMessage().RequestUri.AbsoluteUri;
            currentPage = await page.NextPageRequest.GetAsync(cancellationToken).ConfigureAwait(false);
        }
        while (currentPage is not null);

[LeonarddeR]

Unfortunately, SDK version 5 doesn't have something like a generic ICollectionPage, which makes it impossible to use an extension method like this. That's a real pitty.

[michaelsudnik]

I use the following for IAsyncEnumerable support with SDK version 5, which borrows from the internal concepts used within the out of the box PageIterator type.

public static class CollectionPageExtensions
{
    public static async IAsyncEnumerable<TEntity> AsAsyncEnumerable<TEntity, TCollectionPage>(this TCollectionPage collectionPage, IBaseClient graphClient, [EnumeratorCancellation] CancellationToken cancellationToken = default)
        where TCollectionPage : IBackedModel, IParsable, IAdditionalDataHolder, new()
    {
        TCollectionPage? currentPage = collectionPage;
        do
        {
            var items = currentPage.BackingStore?.Get<List<TEntity>?>("value");

            foreach (TEntity? item in items)
            {
                cancellationToken.ThrowIfCancellationRequested();
                yield return item;
            }

            var nextPage = await currentPage.GetNextPageAsync<TEntity, TCollectionPage>(graphClient, cancellationToken).ConfigureAwait(false);
            if (nextPage == null)
            {
                yield break;
            }

            currentPage = nextPage;
        }
        while (currentPage is not null);
    }

    private static async Task<TCollectionPage?> GetNextPageAsync<TEntity, TCollectionPage>(this TCollectionPage collectionPage, IBaseClient graphClient, CancellationToken cancellationToken = default)
        where TCollectionPage : IBackedModel, IParsable, IAdditionalDataHolder, new()
    {
        var nextPageLink = ExtractNextLinkFromParsable(collectionPage);

        if (string.IsNullOrEmpty(nextPageLink))
        {
            return default;
        }

        // Call the MSGraph API to get the next page of results and set that page as the currentPage.
        var nextPageRequestInformation = new RequestInformation
        {
            HttpMethod = Method.GET,
            UrlTemplate = nextPageLink
        };
        // if we have a request configurator, modify the request as desired then execute it to get the next page
        return await graphClient.RequestAdapter.SendAsync(nextPageRequestInformation, (parseNode) => new TCollectionPage()).ConfigureAwait(false);
    }

    private static string ExtractNextLinkFromParsable<TCollectionPage>(TCollectionPage parsableCollection, string nextLinkPropertyName = "OdataNextLink")
        where TCollectionPage : IBackedModel, IParsable, IAdditionalDataHolder, new()
    {
        var nextLinkProperty = parsableCollection.GetType().GetProperty(nextLinkPropertyName);
        if (nextLinkProperty != null &&
            nextLinkProperty.GetValue(parsableCollection, null) is string nextLinkString
            && !string.IsNullOrEmpty(nextLinkString))
        {
            return nextLinkString;
        }

        if (parsableCollection.AdditionalData == null)
            return string.Empty;

        // the next link property may not be defined in the response schema so we also check its presence in the additional data bag
        if (parsableCollection.AdditionalData.TryGetValue(CoreConstants.OdataInstanceAnnotations.NextLink, out var nextLink))
        {
            if (nextLink != null)
                return nextLink.ToString();
        }
        return string.Empty;
    }
}