Add a mailto to List-Unsubscribe

[jd440]

I think a good feature to help unsubcribe. And since I add it I good a unsubscribe rate from outlook on iphone.

[bramley]

This functionality can probably be implemented in a plugin using the messageHeaders() method. See file admin/defaultplugin.php.

[jd440]

But that's seems for me, that should be a core-feature.

[samtuke]

Making the unsubscribe process easier is a very worthwhile goal. Email headers are primarily read by machines -- won't using HTML in the header content result in the URL being unreadable by automated systems? Are message headers readable by Outlook on iPhone? It is not possible to view them in many mail clients GMail and K9 on Android as far as I'm aware.

[bramley]

@samtuke There isn't any html as such in the header, it is just plain text. @jd440 Please can you explain how this is meant to work? The code is a bit confusing, e.g. is the change intended to add a second unsubscribe header (as suggested by the comment in config_extended.php) or to replace the existing unsubscribe header. Also, some explanation of how the subject and body of the unsubscribe email will be processed.

[jd440]

@bramley

Please can you explain how this is meant to work?

This commit only add a second way to unsubscribe way, on header, This is an addition, and not a substitution So subcriber have a link directly from app/webmail, to unsubcribe, which will send a mail to mailto

Also, some explanation of how the subject and body of the unsubscribe email will be processed.

As you saw, on commit this will send an email to envelope.

+	$unsubscribeMailtoSubject= "Unsubscribe%20By%20Mailto";
+	$unsubscribeMailtoBody = "unsubscribe%20User%20$destinationemail%0D%0AX-MessageId%20:%20$messageid%0D%0Auid=$hash" ;
+	$text["unsubscribeMailto"] = "mailto:" . $GLOBALS['message_envelope'] . "?subject=".$unsubscribeMailtoSubject."&body=" . $unsubscribeMailtoBody . 

Subject : "Unsubscribe By Mailto"; Body : " //the $destinationemail will allow processBounce to identify subscribers unsubscribe User $destinationemail //the X-MessageId will allow processBounce to identify Campaign to automatic process X-MessageId : $messageid //the $hash must improve delivrey by letting webmail identify message as unique uid=$hash" ;

Of course readability of this commit could by improve using urlencode() instead of %20 %0D%0

As discussed elsewhere the value of a setting in config_extended should be the same as the default set in init.php, so false in this case.

Don't worry I will do it, now I understand what you expect on this side (true/false instead 1/0…, init.php) Of course i will be happy to provide a better commit if you interest to merge it. This is only the one that I use for my use.

[bramley]

Of course i will be happy to provide a better commit if you interest to merge it.

Please do that first. I had to manually break this line into pieces to understand what it is doing

+ $mail->addCustomHeader("List-Unsubscribe: ".((isset($text["unsubscribeMailto"]) && $text["unsubscribeMailto"] !="")?$text["unsubscribeMailto"]:"")."<" . $text["jumpoffurl"] . ">"); It is a personal preference but please use variable interpolation or sprintf() when combining several variables and constants like this. Using the "." operator like this becomes unreadable. Why have you used an array $text to hold the intermediate values?

What do you mean by this comment

//the $hash must improve delivrey by letting webmail identify message as unique
uid=$hash" ;

[bramley]

One other comment is about the approach of using the bounce email address.

An email will be sent to that address but to be recognised as a bounce the body of that email must have what appear to be mail headers (X-MessageId and X-ListMember).

@michield Is this approach of spoofing a bounce a good idea?

[jd440]

@bramley I will commit asap, I already done but I have to test:

An email will be sent to that address but to be recognised as a bounce the body of that email must have what appear to be mail headers (X-MessageId and X-ListMember).

I check and test and i does recognise even if is in the body And if we check https://github.com/phpList/phplist3/blob/69052ab76b407f388ce7697f788b7c1df8488ee2/public_html/lists/admin/processbounces.php#L515

$messageid = findMessageId($bounceBody);

is check from body

[jd440]

For information "ist-Unsubscribe header is described in RFC 2369 (for more details please see https://www.ietf.org/rfc/rfc2369.txt). The header field contains either a Web-URL, an email address or both. Since different providers and email clients use that one or the other, we recommend to use both . A List-Unsubscribe header could look like this: List-Unsubscribe: http://www.host.com/list.cgi?cmd=unsub&lst=list, mailto:[email protected]?subject=unsubscribe The most important characteristic of these links is, that they don't require any user interaction by the recipients, so that the unsubscribe is processed by just clicking the link." Source: https://certified-senders.org/wp-content/uploads/2017/07/CSA_list_unsubscribe.pdf

[bramley]

After your previous explanation I looked at http://www.list-unsubscribe.com/ and now see why this is important. My main problem is that I cannot test the process end-to-end because none of my email clients (yahoo, gmail, outlook) display an unsubscribe button.

I was looking at simplifying the code change and came up with this. Note that X-ListMember is the header that is expected by the bounce processing. Only if that is not present does the code look for any email address in the bounce body. Also, rawurlencode() is better than just urlencode().

    $listUnsubscribeHeader = 'List-Unsubscribe: ';

    if (UNSUBSCRIBE_MAILTO) {
        $listUnsubscribeHeader .= sprintf(
            '<mailto:%s?subject=%s&body=%s>, ',
            $GLOBALS['message_envelope'],
            rawurlencode('Unsubscribe by mailto'),
            rawurlencode("X-ListMember: $destinationemail\r\nX-MessageID: $messageid\r\nuid=$hash\r\n")
        );
    }
    $listUnsubscribeHeader .= "<" . $text["jumpoffurl"] . ">";
    $mail->addCustomHeader($listUnsubscribeHeader);

I think that generates the header correctly but I cannot test that the mailto: gets decoded as expected when someone clicks the button

List-Unsubscribe: <mailto:[email protected]?subject=Unsubscribe%20by%20mailto&body=X-ListMember%3A%20foo4%40test.com%0D%0AX-MessageID%3A%2055%0D%0Auid%3Dbb48f785bf004640df01b1e187ca8e88%0D%0A>, <http://strontian/lists/?p=unsubscribe&uid=bb48f785bf004640df01b1e187ca8e88&jo=1>

I think that this change should be tested in the major email clients to ensure that it works correctly in each.

[jd440]

@bramley thanks. Your code seems cleaner, and ok. But i'll test it. If ok i'll commit and PR again.

[jd440]

This version have been test on different webmail and mail provider on mobile and desktop This feature add an "unsubcribe button action" to: outlook app outlook ios mail app laposte.net web

I Just add from the @bramley version in the body mailto, as in some case the unsubscribers could see message I think it's better we could see a message as: "Please Unsubscribe me: " So he could understand whats going on.

Problem

• on laposte.net web, when subscriber click on "unsubscribe button action" the server make a request on : lists/?p=unsubscribe&uid=[…] but as we are not in one-clik unsubscribe, nothing happen.

So I think leave "list-unsubscribe: lists/?p=unsubscribe&uid=" is maybe not a good think as

• User ask to his mail provider to unsubscribe him
• mail provider made a call to lists/?p=unsubscribe
• Fail to unsubcribe
• user still subscribe and mail are sent
• !!!! Bad practise --> decrease ip/domain reputation

[samtuke]

@jd440 Brilliant that it works on Outlook and iOS mail. if only laposte.net has the issue you describe, and the code is technically compliant, then this solution seems acceptable. What do you think?

[michield]

yes, it s a great idea to add this, and I've always had that in mind, except we do need to make sure that the unsubscribes work, otherwise it will be counter productive. Not processing unsubscribe requests will cause backlash, in those situations it's better to not give the impression that you can unsubscribe this way,

So, that means, this PR needs to

1. use the bounce mailbox for the unsubscribe. It may be fine to specify some code in eg the subject to identify the subscriber. It can also be done by detecting the sender, but forwarding and other reasons may mean that the sender is not always the subscriber. Therefore, using VERP or similar would be better, but that will put requirements on the receiving MTA which is not available for most people and would therefore become an advanced feature for self-hosters who own their own server only.

2. tell the bounce process to handle these messages and process the unsubscribe with the necessary audit trail.

Once that's in place, this will be a great addition and I agree it would be good to make it core

[michield]

And just to be clear, I really think we should not add this without those additions. Otherwise we're going to be giving incorrect impressions on the ability to unsubscribe.

The alternative would be that the mailto goes to a person, but from a software developer point of view, we cannot rely on our users to process those requests correctly and timely.

[jd440]

thank @michield for you answer

I totally agree that we must be sure unsubscribe work but i didn't understand your worried.

tell the bounce process to handle these messages and process the unsubscribe with the necessary audit trail.

a simple bounce rule allow to unsubscribe.'

INSERT INTO `phplist_bounceregex` (`id`, `regex`, `action`, `listorder`, `admin`, `comment`, `status`, `count`) VALUES
(320, 'Unsubscribe By Mailto', 'unconfirmuseranddeletebounce', 0, 1, NULL, 'active', 2);

use the bounce mailbox for the unsubscribe. It may be fine to specify some code in eg the subject to identify the subscriber

processbounce identify unsubcribers, and not the sender from the mailto message as subscribers is identify in the body. I did a test : case1

• email1 is subscribe
• email1 receive news
• email1 send "mailto"

--> email1 is unsubcribe

case2

• email1 is subscribe
• email1 receive news
• email1 forward to email2
• email2 send "mailto"

--> email1 is unsubcribe

case3

• email1, & email2 is subscribe
• email1 receive news
• email1 forward to email2
• email2 send "mailto"

--> email1 is unsubcribe

so as you could see initial recipient is always identify. So ok is not perfect as

• someone could unsubcribe someone else
• If in case3 email2 want to unsubcribe himself, from forwarded himself it will not do it.

So we could of course check if sender == bounce , but this will have another side effect. If ;

• email1 & email2 is used by the same user (user1)
• user1 forward||catch all mail from email1 in email2
• user1 want to unsubcribe we MUST to unsubcribe email1 even if email2 is the bounce's sender

[michield]

ah, indeed, using the bounce rule system may work. You example is a test on your system I presume. You'd never write an insert with an ID in the values.

It will still be useful to make that part of the PR, eg by inserting the rule automatically during upgrade.

Yes, the reason we are sending a final "you have been unsubscribed" is exactly to catch the ones where email1 is unsubscribed after forwarding to email2.

[jd440]

what do you mean by:

You'd never write an insert with an ID in the values.

I can add to the PR an insert on upgrade, I'll do it as soon I got some time.

I understand why you send confirmation, by that's again good practise, and already get complaint on confirmation (PR 205)

please check https://litmus.com/blog/the-ultimate-guide-to-list-unsubscribe specially "Not all email clients support list-unsubscrib"

[bramley]

@jd440 @michield Looking at the bounce rule I am bit unsure how reliable this might be. An ordinary bounce will include the headers from the original email, including the list-unsubscribe header. So just looking for the text "unsubscribe by mailto" might match an ordinary bounce, not only the unsubscribe email.

Looking at how the list-unsubscribe header appears in the bounce, the embedded spaces are still %-encoded as %20, Unsubscribe%20by%20mailto so this approach might be ok but seems to be relying on the encoding remaining in place. I have just tested this by manually changing the %20 to spaces and the bounce was matched by the "unsubscribe by mailto" rule.

[bramley]

@jd440 I don't think that the body of the email needs Please Unsubscribe me: $destinationemail\r\n as that does not add anything needed by the bounce processing and is repeating the email address. For comparison look at the unsubscribe header in the Github notification emails.

List-Unsubscribe: <mailto:unsub+003007a87992f05ec3487247a6b997666c440b1288333c1c92cf000000011610b1ee92a169ce0fef5630@reply.github.com>,
 <https://github.com/notifications/unsubscribe/ADAHqGFYkebrRen7ImiI46UARlk_Ui3Bks5sx5XugaJpZM4QBZij>

Also, what is the reason for including the uid value in the body?

[michield]

@bramley good findings. Yes, it'll need to be watertight, and certainly not catch normal bounces.

@jd440 the ID in the table is auto increment, so you exclude it from insert queries

@bramley the Github example uses VERP, which would be the best thing to do, but is less available for most people

[jd440]

@bramley For sure, we have to be sure to not unsubscribe from this rule, But I'm using it for a years and I never false postive. But maybe you can also test it

About body: For sure we can remove. You found example without any body, and I can send you an example with that's not the point. The fact is, when using "unsubcribe action button" from mailto some will display to subscribers the message. So that's why I'm thinking it would be relevant to get a message understandable for users. without any side effect.

About uid; On this point I'm not sure it's really needed. I just read somewhere, of getting a unique id in mailto will help (microsoft, gmail…) to understand this mailto will be use for unsubscribe user with uniq id. But if you want remove it, do it.

@michield About id, I now understand, after reading your answer and my previous post with request. Of course I'm not planning to insert id

[samtuke]

What are the next steps for this PR?

[michield]

I think it needs to be expanded to handle the unsubscription in a reliable way. I think that's quite a lot of work, but then again, if we send out a header saying "Unsubscribe this and this way" and it doesn't work, that would be really bad.

[jd440]

@michield I'm totally agree with you that saying "Unsubscribe this and this way" and it doesn't work, would be worst than anything.

But…

Why to you think this way is unreliable? Did you tried? I'm using it for month now… check and double checking any bounce and i could notice any trouble

Have you thought to an another way to handle it?

[michield]

@jd440 when I look at https://github.com/phpList/phplist3/pull/203/files I don't see any code that handles the unsubscribes. Can you explain how it's done?

[jd440]

merry christmas

Sorry i was busy,

unsucribe is handle by a advanced rule a simple bounce rule allow to unsubscribe.'

INSERT INTO `phplist_bounceregex` (`id`, `regex`, `action`, `listorder`, `admin`, `comment`, `status`, `count`) VALUES
(320, 'Unsubscribe By Mailto', 'unconfirmuseranddeletebounce', 0, 1, NULL, 'active', 2);

[samtuke]

@jd440 Merry Christmas!

Should that bounce rule be included by default in fresh installations?

[michield]

Sorry, I still don't get how that unsubscribes a user reliably. Can you explain?

[jd440]

@samtuke For sure, rule should should be include on upgrade/install and feature available only when advanced rule is activate

@michield once subscriber click on «unsubscribe button» from header/mailto an email will be sent object: "unsubscribe by mailto" content:" Please Unsubscribe me: $destinationemail X-ListMember: $destinationemail X-MessageID: $messageid"

Then Advanced rule process will thread it.