Notifications fail on SMTP providers with throttling

[ctml91]

Describe the bug SMTP rate limiting causes notifications to fail

To Reproduce Steps to reproduce the behavior:

1. Use an email provider that has rate limiting (aka.ms/concurrent_sending)
2. If multiple notifications are processed in one media refresh notifications may fail to send
3. Notifications are not reattempted

Expected behavior Allow adding a delay between notifications, or allow them to be reattempted in a retry queue.

Logs (Logs directory where Ombi is located)

fail: Ombi.Notifications.GenericEmailProvider[0]
      Exception when attempting to send email
      MailKit.Net.Smtp.SmtpCommandException: 4.3.2 STOREDRV.ClientSubmit; sender thread limit exceeded. Visit aka.ms/concurrent_sending for more info. [Hostname=DS7PR10MB4862.namprd10.prod.outlook.com]
         at MailKit.Net.Smtp.SmtpClient.DataAsync(FormatOptions options, MimeMessage message, Int64 size, Boolean doAsync, CancellationToken cancellationToken, ITransferProgress progress)
         at MailKit.Net.Smtp.SmtpClient.SendAsync(FormatOptions options, MimeMessage message, MailboxAddress sender, IList`1 recipients, Boolean doAsync, CancellationToken cancellationToken, ITransferProgress progress)
         at MailKit.Net.Smtp.SmtpClient.SendAsync(FormatOptions options, MimeMessage message, MailboxAddress sender, IList`1 recipients, Boolean doAsync, CancellationToken cancellationToken, ITransferProgress progress)
         at Ombi.Notifications.GenericEmailProvider.Send(MimeMessage message, EmailNotificationSettings settings)

Desktop (please complete the following information):

• OS: CentOS Docker Image

Ombi Version (please complete the following information):

• Version 4.0.1142.0
• Plex Media Server
• Database Type: SQLite

Additional context Add any other context about the problem here.

In the case of microsoft, the limits are below. The main issue I believe is the three concurrent connections.

The service has various limits to prevent abuse and to ensure fair use. An additional limit is being added. Under the new limit, up to three concurrent connections are allowed to send email messages at the same time. If an application tries to send more than three messages at the same time by using multiple connections, each connection will receive the following error message:

432 4.3.2 STOREDRV.ClientSubmit; sender thread limit exceeded

Additional throttling limits for the SMTP Authentication protocol are:

30 messages per minute
Recipient rate limit of 10,000 recipients per day
Exceeding these limits causes the following error:

554 5.2.0 STOREDRV.Submission.Exception:SubmissionQuotaExceededException

[ombi-bot]

Hi!
Thanks for the issue report. Before a real human comes by, please make sure you used our bug report format.
Have you looked at the wiki yet? https://docs.ombi.app/
Before posting make sure you also read our FAQ.
Make the title describe your issue. Having 'not working' or 'I get this bug' for 100 issues, isn't really helpful.
If we need more information or there is some progress we tag the issue or update the tag and keep you updated.
Thanks!
Ombi Bot.