colly
colly copied to clipboard
Queue Depth functionality + Panic with Async
Description
This pull request adds support for Depth in Queue and adds a panic when attempting to use Async with Queue, as they are incompatible. The changes ensure that MaxDepth works as expected when using Queues.
I had to figure out by trial and error that Async caused my Queue based crawler to instantly finish since it thought the returned promises were 'completed' requests.
Changes Made
- Added a default value for
Depth
for Requests made inAddURL
. - Implemented a check in the
AddRequest
method to increase the depth for each nested request. - Added a panic in the
Queue.Run
method to prevent users from using Async with the Queue, as it's not supported.
Checklist
- [x] Added unit tests for change
Please let me know if you require any changes to this Pull Request!
And if there's any interest I did make a UniqueInMemoryQueueStorage
struct in my project that only stores non duplicate entries within the Queue.
Thank you!