apify-client-js icon indicating copy to clipboard operation
apify-client-js copied to clipboard

feat: export PaginationIterator helper class

Open mvolfik opened this issue 3 years ago • 2 comments

mvolfik avatar Sep 07 '22 14:09 mvolfik

wait, i realized this probably isn't really what I wanted. do we have some util for easily iterating over all values from a PaginatedList (e.g. returned by await client.tasks().list()?

mvolfik avatar Sep 07 '22 14:09 mvolfik

Here's what I (well, GitHub Copilot mostly) came up with:

import type { PaginatedList } from 'apify-client';

export async function* iterateList<T>(
    listGetter: (opts: { limit: number; offset: number }) => Promise<PaginatedList<T>>,
    limit: number,
): AsyncGenerator<T, void, never> {
    let offset = 0;
    let total = 0;
    do {
        const {
            items,
            count: returnedCount,
            total: returnedTotal,
        } = await listGetter({ limit, offset });

        offset += returnedCount;
        total = returnedTotal;
        for (const item of items) yield item;
    } while (offset < total);
}

Would it be worth including and exporting this util somewhere?

mvolfik avatar Sep 07 '22 14:09 mvolfik