crawlee
crawlee copied to clipboard
Unify `RequestList` and `RequestProvider` interfaces and extract their "tandem" behavior from `BasicCrawler`
Initial steps have been taken in #2498.
- [ ] Introduce a
IRequestProviderinterface - [ ] Unify
IRequestProviderwithIRequestList - [ ] Introduce a "switching" request provider implementation to better encapsulate the logic of "read from list first, add new requests to queue" currently present in
BasicCrawler - [ ] The transfer from request list to request queue should be done in background, not when fetching requests from the queue
Python implementation here https://github.com/apify/crawlee-python/pull/777