crawler-extension icon indicating copy to clipboard operation
crawler-extension copied to clipboard

Add an option to pause crawling

Open fawaf opened this issue 3 years ago • 8 comments

add option to disable crawling (e.g. the extension) in case of slowof cpu, slow network, other reason, etc.

fawaf avatar Jul 10 '22 09:07 fawaf

Also requested in a review: https://addons.mozilla.org/en-GB/android/addon/mwmbl-web-crawler/reviews/1853830/

daoudclarke avatar Jul 11 '22 22:07 daoudclarke

Since it is likely the user will forget to turn it back on (I know I will) I suggest we allow pausing, but not disabling. We could have two buttons:

  1. Pause for +1h. Pressing the button displays the message "Paused until xx:xx". Pressing repeatedly will increment the number of hours it's paused for.
  2. Resume. Pressing this button removes the message and resumes crawling.

If they want to turn it off completely they can do that through the Firefox extension UI.

daoudclarke avatar Jul 12 '22 08:07 daoudclarke

Behaviour for batches in progress: we need to check if pause has been pressed in the middle of processing a batch. If that is the case, then we should send the batch in its current state (even if there are just a few pages crawled) and stop crawling.

daoudclarke avatar Jul 12 '22 08:07 daoudclarke

err, i meant pausing mainly yeah, heh. maybe allow for a configurable option insead of just a "pause for +1 hour" button? a dropdown or even a direct textbox

fawaf avatar Jul 12 '22 20:07 fawaf

Personally I'd prefer to keep it simple, but I'll leave it to whoever implements it to decide.

daoudclarke avatar Jul 12 '22 20:07 daoudclarke

yeah, please add this, CPU was killing my laptop (i5,16G) recently and had to disable extension manually… Sorry guys

kristianpaul avatar Aug 30 '22 19:08 kristianpaul

It would be great to implement a simple keyboard shortcut to toggle crawling on and off (for exampbe cmd + shift + M).

xplosionmind avatar Jan 21 '23 08:01 xplosionmind

Another use case: disable on metered connections.

benob avatar Sep 19 '23 08:09 benob