Try generate method returning Stream from paginated results

[floydspace]

As requested here, could be good to make streaming api for generated clients

todo:

• investigate if AWS manifest provides some useful pagination info so we can generate streaming api
• consider creating overloaded methods (DynamoDB.scan({}, { stream: true })) or suffixed methods (DynamoDB.scanStream({}))

currently it can be done like this:

//        ┌─── Stream.Stream<Record<string, AttributeValue>, SdkError | ... 4 more ... | ResourceNotFoundError, DynamoDBService>
//        ▼
const streamed = Stream.paginateChunkEffect(
  undefined as Record<string, AttributeValue> | undefined,
  (key) =>
    DynamoDB.scan({
      TableName: "example-table",
      Limit: 25,
      ExclusiveStartKey: key,
    }).pipe(
      Effect.map((result) => [
        Chunk.fromIterable(result.Items!),
        Option.fromNullable(result.LastEvaluatedKey),
      ])
    )
);

[mixja]

AWS SDK provides async iterators for pagination operations, so you can use the following:

import { CloudWatchLogsClient, paginateFilterLogEvents } from '@aws-sdk/client-cloudwatch-logs'

const logs = new CloudWatchLogsClient()

const paginator = paginateFilterLogEvents(
  { client: logs },
  {
    logGroupName,
    filterPattern,
    startTime,
    endTime,
    limit: 500
  }
)

const stream = Stream.fromAsyncIterator(paginator, (error) => { console.error('Error:', error) })

[floydspace]

elegant, thanks for posting it here