crawlee icon indicating copy to clipboard operation
crawlee copied to clipboard

Unify `RequestList` and `RequestProvider` interfaces and extract their "tandem" behavior from `BasicCrawler`

Open janbuchar opened this issue 1 year ago • 1 comments

Initial steps have been taken in #2498.

  • [ ] Introduce a IRequestProvider interface
  • [ ] Unify IRequestProvider with IRequestList
  • [ ] Introduce a "switching" request provider implementation to better encapsulate the logic of "read from list first, add new requests to queue" currently present in BasicCrawler
  • [ ] The transfer from request list to request queue should be done in background, not when fetching requests from the queue

janbuchar avatar May 24 '24 10:05 janbuchar

Python implementation here https://github.com/apify/crawlee-python/pull/777

janbuchar avatar Dec 12 '24 14:12 janbuchar