Adding email support

[jsthomas]

Currently the roadmap has a section about adding support for email. I would like to work on this but need some clarification about the roadmap.

Does it make sense to work on this now? My impression is that a lot of the supporting libraries are in place, and the problem is just to write some simple wrappers around them and "connect everything up". But, I'm not sure how this fits in with other priorities.

What features are "must haves" for a first version of this system? I looked at Django as a reference. In that framework, the email system is described as a "light wrapper" around existing libraries, and basically gives you two functions: send_mail(...) and send_mass_mail(...). Perhaps that could serve as a guide? Alternatively, it looks like the Phoenix docs just provide some examples of how to use a library (bamboo) to send email. Would an example illustrating how to use emile/mrmime/letters together, inside a dream app, be helpful?

Also, should integrations with other providers (SES, mailgun, etc.) be handled in separate libraries? This appears to be how Django does this, e.g. django-ses.

[aantron]

I think all of this should be done in separate libraries, with a few examples here in the Dream repo, as you suggest. Some of what's in the roadmap isn't meant for Dream directly, but needs to exist in the ecosystem in general, for the ecosystem to be useful for Web work. Writing examples is a good way to check whether the support exists, works well, and is easy to use. So, the Phoenix support sounds about right.

A first version of the email system should be usable for authentication emails (email confirmation, password reset, etc.). It should be able to pass spam filters, provided the user (app author) has the right mail provider accounts and setup on their end.

[thangngoc89]

Usually emails are sent via queues. I would say we need this first :)

[dinosaure]

Just to give you some projects about emails:

• mrmime (https://github.com/mirage/mrmime) which is a decoder/encoder of emails
• emile (https://github.com/dinosaure/emile) which parses email addresses
• colombe (https://github.com/mirage/colombe) which is an implementation of SMTP in OCaml
• and we currently working on DKIM/SPF and DMARC

[jsthomas]

Usually emails are sent via queues. I would say we need this first :)

@thangngoc89, this is something I was wondering about too. When building a python web app, it's pretty common to include celery and have task workers available for handling jobs that need to occur in the background (e.g. sending email(s), generating a report, recurring scheduled tasks, etc.).

Is there a standard solution for message queues in OCaml? Is that an important component of putting together a useful email example? I found amqp-client and ocamlmq. I would imagine ocaml-redis might work as well; I've used both redis and rabbitmq for celery.

[thangngoc89]

@jsthomas we are missing a persistent queue solution on OCaml. The backend could be redis/rabbitmq/sql

[jsthomas]

I've been experimenting with building simple email features. I posted a short example here that uses RabbitMQ to queue email jobs; a background worker process pulls tasks from the queue and sends them via Mailgun's API.

Since Mailgun also supports SMTP, I tried using letters as well, but had some difficulties with an exception I don't understand yet. I posted an issue here, hopefully I can get that sorted out and add an SMTP example as well.

[jsthomas]

I've also added a blog post that walks through the example with more explanation and references.

[dinosaure]

The bug about SMTP is fixed, I will try to cut a release as soon as I can.

[aantron]

@jsthomas, thanks for the excellent blog post. I learned a lot from it. It's a great digest of the state of the ecosystem, various alternatives, external services, and a neat example.

I think we can/should do a few things to make it more "visible" from the Dream repo:

1. I'll create a README section for complete, big examples, and link your example from there. @tcoopman's Dream+TEA+Tailwind example can also go there.

2. We might be able to work a pared-down version of the example into the repo itself, if we register an email account, and a trial Mailgun account configured to be able to send only to our email account. Then, it shouldn't be a problem to let people run instances of the example without registering anything at Mailgun themselves. They can basically try out the OCaml side of things without any entanglements. It would be a sort of "email playground." What do you think about this?

3. I'll move some of the library recommendations into the Community projects section (and probably rename it).

If we do (2), we probably need an email provider that has some kind of API that would allow us to delete emails automatically, maybe by API call. It would be annoying to have to log into it to check for any bad spam etc., better just delete it every day or something from a cron job :) I already have the Dream playground dedicated host, which we can run such a job from.

[aantron]

As for the email-sending system, it looks like a pretty straightforward design internally. Each setup needs...

1. A queue. We need to exercise several options to make sure the framework is actually well-designed. I think we should initially support:

   • in-memory queues, for getting started.
   • (maybe) some kind of locally-persisted, brokerless queues, to avoid any Docker or cloud entanglements (as was also probably implied by @yawaramin's suggestion on Discuss). Ideally, we could use or bind something ready-made, rather than writing our own convention for using SQL here (though I've done so for internal projects).
   • Redis, recommending a Docker container. The other option here is to offer RabbitMQ, but for small-to-medium deployments Redis will do, and the advantage is that both the user might be using Redis for other purposes already, and we as the developers will gain more experience with Redis, and will have less to initially support.
   • Amazon SQS, as a typical and popular cloud provider.

2. Senders:

   • Mailgun
   • (maybe) Amazon SES

As I understand it, both Mailgun and SES favor APIs rather than SMTP. It would be good to exercise at least one sender that favors SMTP. Does a good such sender even exist at this point?

We would then need to write good docs on how to contribute additional queue and sender back ends.

A final note: I haven't looked into this at all, but since eml is not constrained to output HTML (it outputs "just strings"), it may be useful in compiled email templates.

I think anything involving Redis, Amazon, etc., should be in separate repos (similar to how Caqti is separate from Dream).

As for the in-Dream API, I think we need to provide guidance to users on how to trigger email rendering asynchronously (such as using Lwt.async, or through a queue), and then, ideally, we would provide some one send function that triggers the whole back end, like in Django (@jsthomas).

The overall integrated email "system" might be separate from Dream as well — but then we need to figure out queueing. Actually, it seems like figuring out how the libraries should be factored into repos is more challenging than the actual system itself, and the only reason for all this unfortunate factoring is to avoid dragging in a ton of dependencies.

[aantron]

So I think for an initial version, we can have some "core" dream-mail library that has a queue interface (signature) and a sender interface (signature), but does not implement them. There will also be dream-mail-memory, dream-mail-redis, dream-mail-sqs for plugging into the queue interface, and dream-mail-mailgun, etc., plugging into the sender interface. When these are linked into a project by the web developer, they will add their module into ref lists inside the base dream-mail.

All these libraries for specific integrations can be in the same dream-mail repo, but as separate opam packages, to separate their potentially extensive dependencies. Maybe dream-mail-memory can come with the core library.

This seems pretty similar to how Caqti works, with its opt-in support for SQLite, MariaDB, and PostgreSQL.

Over time, as we find additional major uses for the queues, we will figure out how to factor them out into their own library.

[aantron]

Also, I think that setup is not Dream-specific, so you don't need the dream- prefix, and feel free to come up with some creative name for this :)

[aantron]

(Although, if we build a Dream-friendly HTTP client library and you choose to use that for accessing APIs, the library will become vaguely Dream-related again, but by that point the requset and response types will be factored out so that it's not necessary to depend on the server to get the client)

[tungd]

I don't think the email interface would need to bundle the queue interface. In fact, I don't think an email interface would be needed at all, given sending email would requires only a few functions to build up the email and actually send it. It would make more sense for such code to belong to a SMTP client library, or a Mailgun client library .etc. What I think really needed here is an easy queuing interface. In the past I've used Lwt.stream + Lwt_stream.n_iter for a simple in-memory queue (it was for a small user-base, so persistent was not needed).

To be fair, I think it's also okay to have an email interface/signature, and having various library implement that signature, similar to other frameworks such as Django.

[aantron]

To be fair, I think it's also okay to have an email interface/signature, and having various library implement that signature, similar to other frameworks such as Django.

That's basically what I'm suggesting. I think people will want to be able to switch services without potentially rewriting their client code, due to different integrations being implemented differently.

I don't think the email interface would need to bundle the queue interface.

I agree with this in principle, and in the long term, but we need to put the queue interface somewhere to start with, and it seems silly to create yet another library when it's not needed as a separate library yet. So I'm suggesting to develop it within the mail library for ease of development, and we can git filter-branch it out later into its own repo.

[jsthomas]

I've written a small library that provides a consistent interface for sending email against different services (currently Mailgun, Sendgrid, and SMTP). Feedback on whether this would be useful to dream users is welcome.

[aantron]

@jsthomas, I just tried the library and was able to land an email (into my spam folder, of course) using its Mailgun example! Thanks!

The library is definitely useful for Dream users. I left some initial feedback in the library issues/PRs.

I'd like to create some kind of "email sandbox," where we have pre-registered Dream-controlled Mailgun, Sendgrid, SES, etc., accounts, configured to send only to a Dream-controlled read-only inbox, which people can view. That way, people can write some test email code without having to register all these accounts just to mess around with the code. I suppose that might be against the terms of some of these services :P I'll have to check.

Does anyone know of an email account provider that can offer easy read-only access to an account inbox?