crawlee icon indicating copy to clipboard operation
crawlee copied to clipboard

Accept any iterable in `RequestProvider.addRequests`

Open janbuchar opened this issue 7 months ago • 1 comments

See https://github.com/apify/crawlee/blob/a228145b583d672fa584d9bd54f1859bd73077af/packages/core/src/storages/request_provider.ts#L195-L195

If we modify this method (and also addRequestsBatched) to accept any iterables, such as generators, we can make bulk insertion of large amounts of requests more memory efficient.

janbuchar avatar May 20 '25 13:05 janbuchar

this might be handy https://github.com/mikro-orm/mikro-orm/blob/master/packages/core/src/utils/Utils.ts#L445-L451

B4nan avatar May 26 '25 11:05 B4nan