colly icon indicating copy to clipboard operation
colly copied to clipboard

Queue Depth functionality + Panic with Async

Open KristinnVikar opened this issue 1 year ago • 0 comments

Description

This pull request adds support for Depth in Queue and adds a panic when attempting to use Async with Queue, as they are incompatible. The changes ensure that MaxDepth works as expected when using Queues.

I had to figure out by trial and error that Async caused my Queue based crawler to instantly finish since it thought the returned promises were 'completed' requests.

Changes Made

  1. Added a default value for Depth for Requests made in AddURL.
  2. Implemented a check in the AddRequest method to increase the depth for each nested request.
  3. Added a panic in the Queue.Run method to prevent users from using Async with the Queue, as it's not supported.

Checklist

  • [x] Added unit tests for change

Please let me know if you require any changes to this Pull Request! And if there's any interest I did make a UniqueInMemoryQueueStorage struct in my project that only stores non duplicate entries within the Queue.

Thank you!

KristinnVikar avatar Jul 20 '23 18:07 KristinnVikar