ArchiveBot icon indicating copy to clipboard operation
ArchiveBot copied to clipboard

Apply delay settings immediately at the start of a job

Open JustAnotherArchivist opened this issue 4 years ago • 0 comments

As I understand it, jobs are currently started without concurrency or delay settings, and those are later set by the settings monitor. This means that a job always starts at 1 concurrency and 0 delay. This may be a problem when targets have heavy rate limiting and even just two or three requests in a short time trigger that.

While setting the concurrency via the command line is broken (https://github.com/ArchiveTeam/wpull/issues/339), delays are fine (--wait), so the pipeline should use that. However, the command line options do not allow for specifying a custom range of delays, so the average of delay_min and delay_max should be used there.

JustAnotherArchivist avatar Jul 10 '21 20:07 JustAnotherArchivist