django-anymail icon indicating copy to clipboard operation
django-anymail copied to clipboard

Add support for Mailtrap

Open cahna opened this issue 1 year ago • 25 comments

Add support for Mailtrap

cahna avatar Nov 10 '24 23:11 cahna

Thanks for this! It looks pretty good at first glance; I'll try to take a closer look later this week.

Mailtrap does not support: inbound emails and reply-to.

Missing inbound support is not unusual; it's fine to just ignore it.

Reply-to is a little surprising. Some ESPs require handling this as an extra header. (I haven't had a chance to look at Mailtrap's docs.)

I am also having trouble getting local builds and tests to work

What's going wrong? (In the contributing docs, I notice the "test a representative combination of Python and Django versions" command is outdated—the current version should be tox -e lint,django51-py312-all,django40-py38-all,docs. But other than that I'd expect it to work. I'm usually on macOS, but the GitHub tests all run on Ubuntu.)

medmunds avatar Nov 12 '24 02:11 medmunds

@medmunds

Reply-to is a little surprising. Some ESPs require handling this as an extra header.

Aha! I'm no email expert, so TIL! I will add the reply-to-via-headers support momentarily.

cahna avatar Nov 17 '24 20:11 cahna

@medmunds

What's going wrong? (In the contributing docs, I notice the "test a representative combination of Python and Django versions" command is outdated—the current version should be tox -e lint,django51-py312-all,django40-py38-all,docs. But other than that I'd expect it to work. I'm usually on macOS, but the GitHub tests all run on Ubuntu.)

Switching to a dev container, using pyenv to install 3.12 and 3.8, and using the updated tox command got further. I got a complaint that python 3.8 was missing, but creating a symlink /usr/bin/python3.8 to the pyenv installed version fixed that. I think I'm good to go for writing tests. Thanks.

cahna avatar Nov 17 '24 20:11 cahna

Thanks for your feedback @medmunds! Taking a look now.

FYI: I only have Sundays available to work on this for the foreseeable future.

cahna avatar Nov 24 '24 20:11 cahna

Searching for "hidden" within the page showed nothing, so I think I saw everything. LMK if I missed something.

cahna avatar Nov 24 '24 21:11 cahna

I will start working on integration tests locally with my own free account. @medmunds, would you be able to setup a free mailtrap account, put the API key and test inbox ID into the repo's github secrets, and expose them to the build as environment variables?

FYI: the free mailtrap tier allows 100 test emails/month.

cahna avatar Nov 24 '24 21:11 cahna

@medmunds I got started on backend unit tests, but I have left several @unittest.skip("TODO: ...") on tests that either might not be relevant for Mailtrap, or that have outstanding questions on how to adjust the backend code to make the test pass.

cahna avatar Nov 30 '24 20:11 cahna

@medmunds, would you be able to setup a free mailtrap account, put the API key and test inbox ID into the repo's github secrets, and expose them to the build as environment variables?

Done: variables ANYMAIL_TEST_MAILTRAP_DOMAIN and ANYMAIL_TEST_MAILTRAP_TEST_INBOX_ID and secret ANYMAIL_TEST_MAILTRAP_API_TOKEN are installed. (Secrets aren't available to PR builds, so the integration tests won't actually run until it's merged into main.)

medmunds avatar Nov 30 '24 23:11 medmunds

Hiya folks!

Is there I could help with to help get this ready?

adamwolf avatar May 06 '25 19:05 adamwolf

@cahna did you still want to work on this? If not, I'll try to finish it up. (Though realistically, probably not until sometime next month.)

@adamwolf thanks for the offer. There are a couple of things that would be helpful:

  • We need to know what Mailtrap's send API response looks like when you try to send to blocked addresses or mixed blocked/not-blocked. (See comment above.)
  • If cahna isn't planning to move forward with this PR, you're welcome to adopt it and continue in a new PR. It would probably be helpful to squash the current commits (maintaining co-author credit for cahna) and continue from there with the unresolved review feedback above.

medmunds avatar May 11 '25 20:05 medmunds

My apologies for ghosting. Work and life has been very busy. Good news is that this has been running in a production environment for 3+ months without any issues (nearing 2 million emails). I want to finish up the integration tests and outstanding comments, but i can't commit to a deadline at this point.

cahna avatar May 11 '25 20:05 cahna

I have a question. For this integration test case:

    def test_all_options(self):
        message = AnymailMessage(
            subject="Anymail Mailtrap all-options integration test",
            body="This is the text body",
            from_email=formataddr(("Test From, with comma", self.from_email)),
            to=[
                "[email protected]",
                'Recipient 2 <[email protected]>',
            ],
            cc=["[email protected]", "Copy 2 <[email protected]>"],
            bcc=["[email protected]", "Blind Copy 2 <[email protected]>"],
            reply_to=[
                '"Reply, with comma" <[email protected]>',
                "[email protected]",
            ],
            headers={"X-Anymail-Test": "value", "X-Anymail-Count": "3"},
            metadata={"meta1": "simple string", "meta2": 2},
            # Mailtrap supports only a single tag/category
            tags=["tag 1"],
            track_clicks=True,
            track_opens=True,
        )
        message.attach("attachment1.txt", "Here is some\ntext for you", "text/plain")
        message.attach("attachment2.csv", "ID,Name\n1,Amy Lina", "text/csv")
        cid = message.attach_inline_image_file(sample_image_path())
        message.attach_alternative(
            "<p><b>HTML:</b> with <a href='http://example.com'>link</a>"
            f"and image: <img src='cid:{cid}'></div>",
            "text/html",
        )

        message.send()
        self.assertEqual(message.anymail_status.status, {"sent"})
        self.assertEqual(
            message.anymail_status.recipients["[email protected]"].status, "sent"
        )
        self.assertEqual(
            message.anymail_status.recipients["[email protected]"].status, "sent"
        )

Mailtrap's sandbox receives this raw email:

MIME-Version: 1.0
Date: Wed, 13 Aug 2025 19:55:17 +0000
Reply-To: "Reply, with comma" <[email protected]>, [email protected]
X-Anymail-Test: value
X-Anymail-Count: 3
Subject: Anymail Mailtrap all-options integration test
From: "Test From, with comma" <[email protected]>
To: [email protected], "Recipient 2" <[email protected]>
Cc: [email protected], "Copy 2" <[email protected]>
Content-Type: multipart/mixed;
 boundary=92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9

--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9
Content-Type: multipart/related;
 boundary=270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5

--270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5
Content-Type: multipart/alternative;
 boundary=326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a

--326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

This is the text body
--326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<p><b>HTML:</b> with <a href=3D'http://example.com'>link</a>and image: <img=
 src=3D'cid:175511491762.9071.11355961105641567026.img@inline'></div>
--326c44ce490a35bf5801d95597ed546386d37f621bf4b57c687b5605597a--

--270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5
Content-Disposition: inline; filename="sample_image.png"
Content-ID: <175511491762.9071.11355961105641567026.img@inline>
Content-Transfer-Encoding: base64
Content-Type: image/png

iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz
AAALEgAACxIB0t1+/AAAABR0RVh0Q3JlYXRpb24gVGltZQAzLzEvMTNoZNRjAAAAHHRFWHRTb2Z0
d2FyZQBBZG9iZSBGaXJld29ya3MgQ1M1cbXjNgAAAZ1JREFUWIXtl7FKA0EQhr+TgIFgo5BXyBUp
fIGksLawUNAXWFFfwCJgBAtfIJFMLXgQn8BSwdpCiPcKAdOIoI2x2Dmyd7kYwXhp9odluX/uZv6d
nZu7DXowxiKZi0IAUHKCvxcsoAIEpST4IawVGb0Hb0BlpcigefACvAAvwAsoTTGGlwwzBAyivLUP
EZrOM10AhGOH2wWugVVlHoAdhJHrPC8DNR0JGsAAQ9mxNzBOMNjS4Qrq69U5EKmf12ywWVsQI4QI
IbCn3Gnmnk7uk1bokfooI7QRDlQIGCdzPwiYh0idtXNs2zq3UqwVEiDcu/R0DVjUnFpItuPSscfA
FXCGSfEAdZ2fVeQ68OjYWwi3ycVvMhABGwgfKXZScHeZ+4c6VzN8FbuYukvOykCs+z8PJ0xqIXYE
d4ALoKlVH2IIgUHWwd/6gNAFPjPcCPvKNTDcYAj1lXzKc7GIRrSZI6yJzcQ+dtV9bD+IkHThBj34
4j9/yYxupaQbXPJLNqsGFgeZ6qwpLP1b4AV4AV5AoKfjpR5OwR6VKwULCAC+AQV4W9Ps4uZQAAAA
AElFTkSuQmCC
--270a8fa0ade9a71b202ef006e9c638d823960b3497f12fa7496396bdf2c5--

--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9
Content-Disposition: attachment; filename="attachment1.txt"
Content-Transfer-Encoding: base64
Content-Type: text/plain

SGVyZSBpcyBzb21lCnRleHQgZm9yIHlvdQ==
--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9
Content-Disposition: attachment; filename="attachment2.csv"
Content-Transfer-Encoding: base64
Content-Type: text/csv

SUQsTmFtZQoxLEFteSBMaW5h
--92f2d5e7accc549534b58d936f37be57c201c16cc2ef5adac662920af5e9--

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

cahna avatar Aug 13 '25 20:08 cahna

For the integration tests of templates, would you please create a template in your mailtrap account?

Needed steps:

  1. Go to templates to add a new template with these details: image
  2. Select "Welcome email" and leave it unchanged: image
  3. Click "Finish": image
  4. Copy the template UUID: image
  5. Add the template UUID to github actions variables as ANYMAIL_TEST_MAILTRAP_TEMPLATE_UUID

cahna avatar Aug 13 '25 20:08 cahna

It appears that bcc testing is not available on their free plan:

image

cahna avatar Aug 13 '25 20:08 cahna

My conscience continues to make me feel bad about not finishing this, so I have added minimal integration tests, added fixes for some of the PR comments, squashed the commits, and rebased on latest main branch.

cahna avatar Aug 13 '25 20:08 cahna

@cahna thanks! I re-rebased to the latest main, so you should pull before any other changes.

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

That sounds correct: if there's only a single message id, it applies to all recipients. I'll try to do some live testing to confirm. (The main goal is have an ID that could be matched with a later status webhook call somehow.)

It appears that bcc testing is not available on their free plan:

That's an interesting pricing differentiator. (Maybe bcc was getting used by spammers? I think there's another ESP who limits bcc recipients to validated domains for that very reason.)

We can just omit bcc from the integration test.

I'd be inclined to label Mailtrap as "Full" support in the feature matrix: we're able to perform integration testing (on everything except bcc) and the 3500 messages/month on the free plan will way more than cover Anymail's test sending.

medmunds avatar Aug 14 '25 21:08 medmunds

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

That sounds correct: if there's only a single message id, it applies to all recipients. I'll try to do some live testing to confirm.

Hmm, If I send to two "to" and two "cc" recipients on a production domain (https://send.api.mailtrap.io/api/send endpoint), I get this response, which seems to mean your zip code is correct:

{
  "success": true,
  "message_ids": [
    "ed1edba0-795c-11f0-0000-f14311a5578e",
    "ed1fc600-795c-11f0-0000-f14311a5578e",
    "ed20fe80-795c-11f0-0000-f14311a5578e",
    "ed21c1d0-795c-11f0-0000-f14311a5578e"
  ]
}

But on a sandbox domain (https://sandbox.api.mailtrap.io/api/send/SANDBOX_ID), I get a single message_id:

{
  "success": true,
  "message_ids": [
    "5040626968"
  ]
}

medmunds avatar Aug 14 '25 22:08 medmunds

the 3500 messages/month on the free plan will way more than cover Anymail's test sending

@medmunds Just a heads-up that the "sandbox" is only limited to 100/month

cahna avatar Aug 15 '25 01:08 cahna

The parsed_response from Mailtrap's API contains:

{'success': True, 'message_ids': ['5038769279']}

I suppose this means that all of the to, cc, and bcc addresses should receive the same AnymailRecipientStatus? (this means the zip code I have doesn't work and needs fixing)

That sounds correct: if there's only a single message id, it applies to all recipients. I'll try to do some live testing to confirm.

Hmm, If I send to two "to" and two "cc" recipients on a production domain (https://send.api.mailtrap.io/api/send endpoint), I get this response, which seems to mean your zip code is correct:

{
  "success": true,
  "message_ids": [
    "ed1edba0-795c-11f0-0000-f14311a5578e",
    "ed1fc600-795c-11f0-0000-f14311a5578e",
    "ed20fe80-795c-11f0-0000-f14311a5578e",
    "ed21c1d0-795c-11f0-0000-f14311a5578e"
  ]
}

But on a sandbox domain (https://sandbox.api.mailtrap.io/api/send/SANDBOX_ID), I get a single message_id:

{
  "success": true,
  "message_ids": [
    "5040626968"
  ]
}

Hmm... strange behavior. What do you think would be best to do here? If self.testing_enabled expect one ID and use it for all recipients, but if not, use the zip code?

I have only been using Anymail to send single-recipient emails, so this is not something I had encountered, yet.

cahna avatar Aug 15 '25 01:08 cahna

would you please create a template in your mailtrap account?

Done

medmunds avatar Aug 15 '25 18:08 medmunds

What do you think would be best to do here? If self.testing_enabled expect one ID and use it for all recipients, but if not, use the zip code?

That seems reasonable. Maybe check that the length is as expected (1 for sandbox, # recipients for production), just in case they change an API later.

I suspect the difference is that the sandbox records one "message" per API call, however many recipients it has, and the message id is meant to retrieve that single sandbox record. But production email must actually send a separate message to each recipient, and uses individual message ids so you can match a tracking event to its recipient. (Django's mail test outbox has a similar distinction.)

medmunds avatar Aug 15 '25 19:08 medmunds

the 3500 messages/month on the free plan will way more than cover Anymail's test sending

@medmunds Just a heads-up that the "sandbox" is only limited to 100/month

Oh, hadn't noticed that, thanks.

I think the math is probably still OK. The sandbox is 100 send API calls, with no limit on number of "recipients" per send. Production is 3500 messages sent, where each recipient counts as a separate message. (It's the multiple recipients on something like test_all_options that eats through some ESP's smaller quotas. But if we start bumping into the sandbox limit maybe we investigate combining sandbox integration tests.)

While I'm thinking about sandbox vs production:

  • I suspect we should have separate (but similar) integration tests for sandbox and production. They're really two different APIs.
  • I'm wondering if it would make sense to combine the MAILTRAP_TESTING and MAILTRAP_TEST_INBOX_ID settings? Or really eliminate the separate "testing" one. If test_inbox_id is set, that implies testing_enabled. 🤔

medmunds avatar Aug 15 '25 19:08 medmunds

(Hey GitHub: this PR closes #345.)

medmunds avatar Aug 15 '25 19:08 medmunds

@cahna I rebased your original commit and then added a few changes per the discussions above.

I'll try to move this forward over the weekend or early next week. I think the primary remaining work is:

  • [ ] Use Mailtrap's batch APIs with Anymail's merge_data and merge_metadata
  • [ ] Make sure the backend tests aren't missing any cases (ones we usually cover with other ESPs, plus some Mailtrap-specific logic around sandbox vs transactional APIs)
  • [ ] Update the integration tests to cover both sandbox and transactional APIs
  • [ ] Finish the docs

Also, I de-emphasized support for Mailtrap's bulk API in the docs, because I couldn't tell whether it was really the same API payloads/responses as transactional just at a different endpoint. (And didn't want to add to the testing burden if not.)

medmunds avatar Oct 17 '25 02:10 medmunds