Scheduler rate limit functionality

[jpbruinsslot]

Changes

This PR adds rate-limiting capabilities to the scheduler for tasks. Boefjes that are subject to rate limiting (defined in the boefje manifest) are being tracked by an in-memory rate limiter leveraging the package limits. When a task is subject to a rate-limit it will get the DELAYED status and will be postponed until it is allowed again. A thread will check for delayed tasks and put them on the queue when they're ready.

This will expect some changes in the katalogus mainly exposing a grouping/namespacing on which to rate limit and the parseable rate limit.

Issue link

Closes #1317

[jpbruinsslot]

@ammar92 I've implemented cross-organisational support would you mind taking a look?

[praseodym]

Originally posted at https://github.com/minvws/nl-kat-coordination/pull/1413#pullrequestreview-1569235551, but still valid:

As far as I understand, the rate limiter is a property of the BoefjeScheduler class. This class has a separate rate_limiter instance per organisation, which means that rate limiting only works per organisation.

In larger deployments, API keys such as for Shodan will likely be shared by multiple KAT organisations. How do we implement rate limiting for that scenario?

[jpbruinsslot]

Originally posted at #1413 (review), but still valid:

As far as I understand, the rate limiter is a property of the BoefjeScheduler class. This class has a separate rate_limiter instance per organisation, which means that rate limiting only works per organisation.

In larger deployments, API keys such as for Shodan will likely be shared by multiple KAT organisations. How do we implement rate limiting for that scenario?

I've updated the code so that the rate limiter memory is now on the application level. The identifier of the rate limiter can then be formatted with a key for instance so that it will be shared across multiple organisations.

[underdarknl]

In larger deployments, API keys such as for Shodan will likely be shared by multiple KAT organisations. How do we implement rate limiting for that scenario?

This also means that we should probably include both 'shodan' and the actual key in the 'group' to make sure we don't limit shodan requests across the whole install even though we might have multiple keys available.