listmonk doesn't track send errors properly

[hach-que]

Version:

• listmonk: v2.3.0
• OS: Docker/Kubernetes

Description of the bug and steps to reproduce:

Listmonk reports that it's sent 1072 messages:

but in Mailgun I only see 113 messages as sent. When I check the listmonk logs, there's a ton of:

subscriber 2622d138-6406-4a53-8ebe-7403ba43df6d: 503 Only one sender per message, please

and

subscriber 9d4a297d-f264-4dd3-9ab5-d00181d8b427: 421 Domain ... is not allowed to send: request limit exceeded, try again after Tue, 21 Mar 2023 08:38:42 UTC

Because listmonk isn't handling SMTP errors properly, it thinks it's successfully sent out emails that are never actually sent. Worse still, I can't easily re-send only to the failures without manually pulling the logs out of Mailgun and generating a new list to send to.

[hach-que]

I managed to workaround this for now by setting very low concurrency limits:

And I used this script to generate a new list to import into Listmonk for the re-send:

param ($ApiKey, $InputSubscribers, $Subject, $Path, $Domain)

$ExistingSubscribers = @{}
foreach ($Line in (Get-Content $InputSubscribers)) {
    $S = $Line.Split(",")
    if ($S[0] -eq "uuid") {
        continue
    }
    $ExistingSubscribers[$S[1]] = $S[2]
}

$Excluded = 0

$Url = "https://api.mailgun.net/v3/$Domain/events"
do {
    $Content = $(curl.exe -s --user "api:$ApiKey" -G $Url | Out-String)
    $Json = $Content | Convertfrom-Json
    $Url = $Json.paging.next

    foreach ($Entry in $Json.items) {
        if ($Entry.message.headers.subject -eq $Subject) {
            if ($ExistingSubscribers[$Entry.recipient] -ne $null) {
                $ExistingSubscribers[$Entry.recipient] = $null
                $Excluded += 1
            }
        }
    }

    if ($Json.items.Length -eq 0) {
        break
    }
} while ($Url -ne $null)

$List = @("email,name,attributes")
$ExistingSubscribers.Keys | % {
    if ($ExistingSubscribers[$_] -ne $null) {
        $List += "`"$_`",`"$($ExistingSubscribers[$_])`",`"`""
    }
}

Set-Content -Path $Path -Value $List

Write-Host "Excluded $Excluded people from send list"

The way you use this is like so:

.\create-unsent-list.ps1 -ApiKey "mailgun API key" -Path export_path_for_new_list.csv -Subject "subject of email you sent" -InputSubscribers "exported list of subscribers from listmonk" -Domain "domainyousentfrom.com"

[knadh]

Sent refers to e-mails sent to one or more asynchronous worker queues, and not to SMTP servers. Errors are tracked and campaigns are auto-paused when hitting the configured threshold.

I agree that this is confusing. Have to figure out a way to incorporate the error count to sent.

[hach-que]

In that case I'd rename Sent to something like "Queued" and have a separate counter that actually measures how many emails have been successfully sent and have errored out of the total.

There should also be an easy way to re-queue failed sends so you don't have to:

• Extract the list out of the mail server
• Negate it with the existing subscriber list to generate the failure list
• Create a new private subscriber list and import the failure list
• Clone the campaign and send it against the failure list
• Repeat these steps until there are no more failures in the logs

[knadh]

Agree. There should be a retry feature, which is completely absent currently.

[K1D77A]

Just had to reduce my concurrency and message rate to 1 in order to send through mailgun correctly. Listmonk would say 202 While Mailgun would say 26

Reducing the mail send rate fixed this.

[The-Real-Thisas]

Ideally it should at least show some error like 'some emails have failed to deliver', and thanks for the temp fix @hach-que

[The-Real-Thisas]

Also the resent is a good idea @knadh , I really really wish I had it now, I'm digging logs at nearly 1 am to manually send out emails that were timed out or failed to send.

[The-Real-Thisas]

@hach-que I think the title should be changed to reflect the issue at hand "listmonk doesn't display errors in frontend" since as @knadh said it does infact know when an email has failed and will pause after a set number of failures.

[knadh]

The errors are logged async in Settings -> Logs.

A new feature was merged last week (https://github.com/knadh/listmonk/commit/a2da75ce9866cc8c75343c5fc2e8a1e706c1f754) that streams errors to the UI where they popup as alerts (if the UI is open at that moment).

[hach-que]

@The-Real-Thisas The issue is that "Sent" isn't reflecting the detected errors, because the number is only how many emails were pushed to the background workers and not how many emails were successfully sent by the background workers. Whether they display elsewhere in the UI isn't really helpful (especially since the logs show a subscriber ID but that doesn't really help).

An even better solution than only updating "Sent" to reflect successful sends would be to have the workers automatically re-queue emails that failed to send (up to X retries) and record which specific subscribers could not have emails sent to them. A campaign should never count as "complete" unless all subscribers were successfully sent an email (and the mail server didn't return an error); i.e. I should always be able to press "continue" to retry the subscribers that were failed in case I need to fix up my mail server and give it another go (in Mailgun's case, I had to go through verification to increase the sending limit).

[keybits]

Perhaps a better solution is to send via the API of the provider (where available). For Mailgun it will be 3x faster and allows batch sending: https://documentation.mailgun.com/en/latest/quickstart-sending.html#send-with-smtp-or-api

Would you accept a PR for sending via the API of Mailgun?

[knadh]

The issue is that "Sent" isn't reflecting the detected errors, because the number is only how many emails were pushed to the background workers and not how many emails were successfully sent by the background workers. Whether they display elsewhere in the UI isn't really helpful (especially since the logs show a subscriber ID but that doesn't really help).

This is now addressed. It involved a full rewrite of the core. https://github.com/knadh/listmonk/commit/53f5b7dc6654a5a272107c56ff3f41b4cc02f82a