crawlee
crawlee copied to clipboard
Accept any iterable in `RequestProvider.addRequests`
See https://github.com/apify/crawlee/blob/a228145b583d672fa584d9bd54f1859bd73077af/packages/core/src/storages/request_provider.ts#L195-L195
If we modify this method (and also addRequestsBatched) to accept any iterables, such as generators, we can make bulk insertion of large amounts of requests more memory efficient.
this might be handy https://github.com/mikro-orm/mikro-orm/blob/master/packages/core/src/utils/Utils.ts#L445-L451