Bounced email processing

[hoyes]

Bounced e-mails are currently sent to websupport@ (when an e-mail we send out is rejected by the recipient's mailserver):

We should:

1. Set the return path correctly so bounced e-mails end up at a different address
2. Log the contents of the bounced e-mail somehow
3. Attempt to parse the bounce e-mail and set "is_email_verified" to false on the relevant user so we don't attempt to send anything to that address again.

Potential difficulties in automatic parsing:

• I'm guessing the format is non-standard. Probably have to parse the body for e-mail-like strings
• Vacation auto-responders and "mailbox full" messages may look similar to bounces but we don't necessarily want to mark the e-mail address as invalid in these cases Maybe need to do some research into whether there are standard ways of approaching the above.

[hoyes]

Example (redacted) bounced e-mail from Cambridge UCS:

This is a MIME-encapsulated message.

--ED5E8101B4.1518995165
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii

This is the mail system at [hostname]

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The mail system

: host mx.cam.ac.uk[131.111.8.146] said: 550-
is not a known user on this system; 550 see
http://help.uis.cam.ac.uk/email-bounce (in reply to RCPT TO command)

--ED5E8101B4.1518995165/[hostname]
Content-Description: Delivery report
Content-Type: message/delivery-status

Reporting-MTA: dns; mail.[hostname]
X-Postfix-Queue-ID: ED5E8101B4
X-Postfix-Sender: rfc822; [email protected]
Arrival-Date: Sun, 18 Feb 2018 23:06:03 +0000 (GMT)

Final-Recipient: rfc822; [email address]
Original-Recipient: rfc822;[email address]
Action: failed
Status: 5.0.0
Remote-MTA: dns; mx.cam.ac.uk
Diagnostic-Code: smtp; 550- is not a known user on this
system; 550 see http://help.uis.cam.ac.uk/email-bounce

--ED5E8101B4.1518995165/[hostname]
Content-Description: Undelivered Message
Content-Type: message/rfc822

Received: from [hostname] (localhost [127.0.0.1])
by [hostname] (Postfix) with ESMTP id ED5E8101B4
for ; Sun, 18 Feb 2018 23:06:03 +0000 (GMT)
Received: from tiberius.camdram.net (tiberius.camdram.net [5.172.155.149])
by [hostname] (Postfix) with ESMTP id 654C6102EF
for ; Sun, 18 Feb 2018 23:06:02 +0000 (GMT)
Received: by [hostname] (Postfix, from userid 33)
id EAF5D3A6A6; Sun, 18 Feb 2018 23:06:01 +0000 (GMT)
To: [name] <[email]>
Subject: Camdram update
X-PHP-Originating-Script: 1008:SimpleMailInvoker.php
Message-ID: 
Date: Sun, 18 Feb 2018 23:05:59 +0000
From: Camdram 
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Swift-Return-Path: 
X-Swift-Return-Path: 

Lorem ipsum

--ED5E324273840324--

[philosophicles]

I think UCS bounces form the vast majority of the ones we receive, so coding those as a automated special case seems sensible. Note though they do sometimes include a section of text like "try this email address instead: [email protected]" as graduating students can specify a new email of their choice.

I'd hesitate to implement a fully automated solution more generally. For non-UCS sources, situations like the one above might cause us to pick a valid user email address from within the bounce notification and invalidate it. For non-UCS bounces, I feel like an admin review queue would be a better 'full-featured' way to approach this - parse out the likely email address (check it's actually associated with a user), then add to a queue where the human looks at the full bounce text, and the probable email, and clicks 'agree' to cause it to be marked invalid. Easier, halfway house approaches for non-UCS bounces would also be fine.

[philosophicles]

A solution that automatically discards out-of-office / vacation notifications would also be useful, as we receive a lot of those at certain times of year!

[CHTJonas]

I've added a few filtering rules to Brimir (see https://support.camdram.net/rules) to help deal with this.

Is the bounce address set with a specific header in the email itself? My thinking is that we could add a header that specifies a bounce address of eg. [email protected] and create an associated mailbounces user (or a simple vhost redirect) to store them in an mbox on Tiberius rather than creating spurious support tickets.

[hoyes]

Just done a little more research into this. My understanding is that the Return-Path header can be set to specified to determine where the e-mail should be returned to if it cannot be delivered. I don't think there's a way to set this header globally in PHP, so might be better to configure this Postfix.

https://en.wikipedia.org/wiki/Variable_envelope_return_path looks like an interesting technique to implement it. The idea is that we set the Return-Path to [email protected], and pipe anything that lands there to an handler which parses the email address to obtain the e-mail to un-verify.

https://tools.ietf.org/html/rfc3834 and https://github.com/jpmckinney/multi_mail/wiki/Detecting-autoresponders has some interesting stuff about auto-responders. I think this is only really relevant for ADC management for us, though so it might be sufficient to just inspect the headers of those messages and add whatever custom logic is necessary for them.

[CHTJonas]

As a really simple solution, we could add a line to the brimir user procmail file that matches subjects or bodies containing "undeliverable" or "returned to sender" and process them that way?

[hoyes]

Trying to avoid touching the camdram DB from outside this codebase, but could maybe just make the console command:

app/console camdram:unverify-email [email protected]

which handles setting the flag only, then do all the bounce logic externally in procmail etc

[hoyes]

Just realised after I sent this:

There are probably two halves of this:

• Bounced e-mails sent by Camdram code (notification e-mails and mailouts etc)
• Bounced e-mails sent by Brimir

Which may or may not have different solutions...

[philosophicles]

How often are emails sent by Brimir going to bounce? Very infrequently I think: I think this will only happen if somebody emails [email protected] with a problem, and by the time we get to reply, their email has ceased to exist. Those rare cases might best be handled by just showing them in Brimir like any other email / reply.

For me this issue is pretty much exclusively about emails that originate as transactional notifications sent by Camdram code - your first bullet point.

[hoyes]

Yea I agree, but I guess it's slightly confusing because bounced transactional notifications currently end up in Brimir...

But yes, it's probably fine (and maybe even desirable) for bounced support e-mails to end up back in the same support thread. The original purpose of this ticket though was about connecting the return path of transactional e-mails to the "is_email_verified" flag on the corresponding user.