Proposal: add per-provider rate limit

[ajoslin]

I use multiple SMTP providers, and we are currently adding a couple. As we warm these providers up, would like to set a daily limit for these providers, so that when the limit is hit it will move to another one.

For example: new SES account at 50,000 per day.

Fixes #1097

[ajoslin]

This needs cleanup and i18n - this is only to check for interest from @knadh.

[knadh]

Hi @ajoslin.

• How does this interact with the global sliding window limit?
• A daily limit isn't universal. If at all, it should be a custom time period. X messages per Y duration, like the sliding rate limit.

[halms]

I think that a per-provider rate limit is a very good enhancement. I'd say this should be implemented like the sliding rate limit (maybe also allowing 'per day') as the window.

From my point of view, this could replace the global sliding window limit, Maybe I'm missing something, but I'd say the typical use case for the sliding window is to obey provider restrictions. But I'm happy to be convinced otherwise.

What I'd personally need is a different limit for Emails (SMTP) and other messengers (where I'm working to implement a custom backend around signald and whatsmeow). A limit per provider (SMTP servers and messengers) would fulfill this very nicely.

[ajoslin]

• How does this interact with the global sliding window limit?

This is about each individual provider. For example, right now we have a new plan with a new SMTP provider at a great deal but they require a long 60 day warmup period with a new quota daily. Each day I go in and edit the daily quota.

In the meantime, we continue to allow other providers to fill in the remaining messages sent daily.

• A daily limit isn't universal. If at all, it should be a custom time period. X messages per Y duration, like the sliding rate limit.

Sure, the duration could be configurable. I like that.

---

I modified this to use https://github.com/go-redis/redis_rate and things still send fast enough to 1M+ subscribers.

Bad part is it adds another dependency to the stack, but I think it could be coded in such a way that the Redis connection is optional.

I also added an escape hatch when there are no providers available to throw an error too, which will pause the campaign.

I'll upload it soon to this PR since it seems to be getting good feedback.

[knadh]

Closing this as there are unresolved issues and master has moved ahead.