crawlee icon indicating copy to clipboard operation
crawlee copied to clipboard

Add `waitForAllRequestsToBeAdded` option to `enqueueLinks`

Open barjin opened this issue 1 year ago • 0 comments

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/core

Feature

As discussed in this WCC PR, the enqueueLinks method (and the contexted version too) is missing the waitForAllRequestsToBeAdded option, which means that the method always resolves after enqueuing 1000 URLs, with no opt-in for waiting until the end.

cc @vladfrangu as discussed :)

Motivation

Some (high-concurrency?) use cases could profit from this.

Ideal solution or implementation, and any additional constraints

Should be quite straightforward to implement (just pass the new option to the RQ).

Alternative solutions or implementations

No response

Other context

No response

barjin avatar Feb 02 '24 17:02 barjin