crawlee
crawlee copied to clipboard
Add `waitForAllRequestsToBeAdded` option to `enqueueLinks`
Which package is the feature request for? If unsure which one to select, leave blank
@crawlee/core
Feature
As discussed in this WCC PR, the enqueueLinks method (and the contexted version too) is missing the waitForAllRequestsToBeAdded option, which means that the method always resolves after enqueuing 1000 URLs, with no opt-in for waiting until the end.
cc @vladfrangu as discussed :)
Motivation
Some (high-concurrency?) use cases could profit from this.
Ideal solution or implementation, and any additional constraints
Should be quite straightforward to implement (just pass the new option to the RQ).
Alternative solutions or implementations
No response
Other context
No response