SMTP: TLS + port 465 doesn't work

[LyricSeraph]

Describe the bug Sending email with SMTP not work.

To Reproduce Steps to reproduce the behavior: Using the docker compose version to self-host a asciinema server. Setting the SMTP_* environments below:

- SMTP_HOST=smtp.qq.com
- SMTP_PORT=465
- SMTP_AUTH=always
- SMTP_USERNAME=**MY_EMAIL_ADDRESS**
- SMTP_PASSWORD=**MY_PASSWORD**
- SMTP_TLS=always
- SMTP_FROM_ADDRESS=**MY_EMAIL_ADDRESS**

Then send with the test email command:

docker-compose exec asciinema send-test-email MY_EMAIL_ADDRESS

The command is freeze without any output.

The signup page triggers a Asciinema.Emails.Job with the log below:

asciinema-1  | 11:36:21.151 [info] {"args":{"to":"[email protected]","type":"signup","url":"http://MY_HOST_NAME/users/new?t=SFMyNTY.g2gDbQAAABVseXJpY2xhd0BseXJpY2xhdy50b3BuBgDgvroKjwFiAAFRgA.9pvVT2YUvleHVixnwODV0bKMnai1StUZgkHz4qFMvZc"},"attempt":3,"duration":60106026,"error":"** (ArgumentError) raise/1 and reraise/2 expect a module name, string or exception as the first argument, got: {:network_failure, '**MY_IP_ADDRESS**', {:error, :closed}}","event":"job:exception","id":6,"max_attempts":20,"meta":{},"queue":"emails","queue_time":137879,"source":"oban","state":"failure","tags":[],"worker":"Asciinema.Emails.Job"}

After formatted:

{
  "args": {
    "to": "[email protected]",
    "type": "signup",
    "url": "http://MY_HOST_NAME/users/new?t=SFMyNTY.g2gDbQAAABVseXJpY2xhd0BseXJpY2xhdy50b3BuBgDgvroKjwFiAAFRgA.9pvVT2YUvleHVixnwODV0bKMnai1StUZgkHz4qFMvZc"
  },
  "attempt": 3,
  "duration": 60106026,
  "error": "** (ArgumentError) raise/1 and reraise/2 expect a module name, string or exception as the first argument, got: {:network_failure, '**MY_IP_ADDRESS**', {:error, :closed}}",
  "event": "job:exception",
  "id": 6,
  "max_attempts": 20,
  "meta": {},
  "queue": "emails",
  "queue_time": 137879,
  "source": "oban",
  "state": "failure",
  "tags": [],
  "worker": "Asciinema.Emails.Job"
}

The error message maybe indicated that there are some bugs in the source code.

Expected behavior The server should send the email as expected.

Screenshots The screenshot of the above log:

Versions:

• OS: Ubuntu 20.04 LTS x86_64(5.4.0-77-generic), Arch Linux x86_64(6.8.7-arch1-1)
• Browser: Chrome
• asciinema server: docker image: ghcr.io/asciinema/asciinema-server:20240324 hash: c0ea6c90a8e4

Additional context The SMTP configs are verified on both thunderbird client and online wordpress SMTP plugin, which means the SMTP settings should be correct and the issue is not relative to the firewall.

[ku1ik]

The error includes {:network_failure, '**MY_IP_ADDRESS**', {:error, :closed}}, which suggests a network connection error.

Maybe the connection is closed by the SMTP server due to incompatible TLS version?

Have you tried setting SMTP_ALLOWED_TLS_VERSIONS to tlsv1.1 or tlsv1.2 explicitly?

[LyricSeraph]

It seems the network failure is due to the misspelling on my password. But after I corrected the password, the email job just has no response. The job just leaves an "attempt: 1" on Email.Job and no further related logs were generated. The mailserver backend shows me that there's no login activity for the new generated token.

And still, I have tried specifying the SMTP_ALLOWED_TLS_VERSIONS with any combination of tlsv1, tlsv1.1, tlsv1.2 and even tlsv1.3.

I am trying to build the server with debug messages enabled. I will comment later if I found any new updates.

[jiriks74]

Try to remove

- SMTP_PORT=465
- SMTP_TLS=always
- SMTP_AUTH=always

I had issues with these options. After deleting them, leaving only these:

SMTP_HOST=<my-mailserver>
SMTP_USERNAME=<login-mail>
SMTP_FROM_ADDRESS=<login-mail>
SMTP_REPLY_TO_ADDRESS=<admin-mail>
SMTP_PASSWORD=<pass>

I got everything to work. IDK why but tls+465 just didn't work with asciinema + my mail server. It could work as a temporary fix but it still should be investigated why these options don't seem to be working.

[LyricSeraph]

Try to remove

- SMTP_PORT=465
- SMTP_TLS=always
- SMTP_AUTH=always

I had issues with these options. After deleting them, leaving only these:

SMTP_HOST=<my-mailserver>
SMTP_USERNAME=<login-mail>
SMTP_FROM_ADDRESS=<login-mail>
SMTP_REPLY_TO_ADDRESS=<admin-mail>
SMTP_PASSWORD=<pass>

I got everything to work. IDK why but tls+465 just didn't work with asciinema + my mail server. It could work as a temporary fix but it still should be investigated why these options don't seem to be working.

OMG IT WORKS!!!

I have spent my whole weekend last week diving into the code. But I still can't find out what's wrong in the config. The STMP client (Bamboo) just fired a log about the mail content and hung. Maybe there's something problem related to the Boomboo SMTP client itself. I almost decide to modify the code to send emails externally.

You save my time!

But I just remove the line SMTP_PORT, which means it will use the predefined port 587 in code. And the SMTP_TLS can't be removed in my case as my mailiing server froce the usage of SSL.

ANYWAY, THANK YOU SO MUCH

[LyricSeraph]

@ku1ik The issue is solved in my case. If you think this can be accepted as a temporary solution, you can close the issue.

[jiriks74]

I am not sure how I solved it but I think that my mail server reported some login issues and that's why I tried it this way. I may look into it again it that's something @ku1ik wants.

ANYWAY, THANK YOU SO MUCH

You're welcome, glad I could help <3

[ku1ik]

Thanks for help @jiriks74 :raised_hands:

I'm not sure why wouldn't TLS + 465 work. The configuration is defined here, and all the options are passed directly to Bamboo's SMTP adapter: https://github.com/asciinema/asciinema-server/blob/b3a852d769da24b1360562b443e30ca41de23351/config/runtime.exs#L144-L177

But, since both of you got it working I'm going to close this issue.

[jiriks74]

@ku1ik I don't think that this should be closed as the tls+465 option still doesn't work. In this case I'd consider the solution a workaround as we didn't fix anything we just used something else to make it work until we figure out what's wrong.

[ku1ik]

Fine by me :+1: Feel free to investigate then :)

[ku1ik]

Since I've upgraded the SMTP client library in the today's release you may want to try it out. There's a slight chance it fixed the issue.

[ku1ik]

Also, I believe there's a difference between using TLS and using SSL... Maybe what you need is enabling SSL mode?

The SMTP client we use allows enabling both, separately, so there is a distinction. Right now SSL is always disabled, and there's no env var to enable it, but we could add one.

However, SSL mode can also be enabled today by using a custom config file and putting this in it:

import Config
config :asciinema, Asciinema.Emails.Mailer, ssl: true

This will work in addition to the existing env vars, although it's probably wise to remove STMP_TLS when enabling SSL as above.

[jiriks74]

Using

- SMTP_PORT=465
- SMTP_TLS=always

results in 502 when logging in for me.

[jiriks74]

@ku1ik

Also, I believe there's a difference between using TLS and using SSL... Maybe what you need is enabling SSL mode?

There's a difference but you could say that TLS is basically SSLv2?

TLS builds on the now-deprecated SSL (Secure Sockets Layer) specifications (1994, 1995, 1996) developed by Netscape Communications for adding the HTTPS protocol to their Netscape Navigator web browser.

According to this you don't want to use SSL as it's quite old.

[ku1ik]

Right. At the client library level (both the previous one and the new one I switched to) there's distinction, there are 2 separate options to enable/disable those, named tls and ssl. The ssl one defaults to false, and tls defaults to if_available. I haven't exposed the ssl option via env var because, as you said, it's old and nobody should use it anymore (and there's an escape hatch for those who need it through the advanced Elixir config file). I only mentioned it in my previous comment to rule out the small possibility that you or @LyricSeraph needed it, but now it's pretty clear we're all after TLS :)

[ku1ik]

results in 502 when logging in for me.

502? As in Bad Gateway error?

Is there anything interesting in the container logs when this happens?

[jiriks74]

502? As in Bad Gateway error?

Yup, it's 502 indeed.

[ku1ik]

This is from Cloudflare, which I assume you put in front of asciinema server.

Can you pull container logs from the moment this error happened?

[jiriks74]

This is from Cloudflare, which I assume you put in front of asciinema server.

Yup but this issue is caused by Asciinema. The only thing that Cloudflare does is make a cool site for the error.

Nothing interesting in logs

Logs

19:07:36.286 request_id=F9NuBkw-4OHVNR4AAADh [info] GET /login/new

19:07:36.288 request_id=F9NuBkw-4OHVNR4AAADh [info] Sent 200 in 2ms

19:07:38.993 request_id=F9NuBu2aFl8Z7cQAAAEB [info] POST /login

19:08:00.626 [info] {"args":{},"attempt":1,"event":"job:start","id":25953,"max_attempts":20,"meta":{},"queue":"default","source":"oban","system_time":1716836880625784378,"tags":[],"worker":"Asciinema.Streaming.GC"}

19:08:00.636 [info] {"args":{},"attempt":1,"duration":4671,"event":"job:stop","id":25953,"max_attempts":20,"meta":{},"queue":"default","queue_time":26763,"source":"oban","state":"success","tags":[],"worker":"Asciinema.Streaming.GC"}

19:08:39.902 request_id=F9NuFRwKVOYfMgQAAAIB [info] GET /favicon.ico

19:08:39.903 request_id=F9NuFRwKVOYfMgQAAAIB [info] Sent 404 in 920µs

19:09:00.646 [info] {"args":{},"attempt":1,"event":"job:start","id":25954,"max_attempts":20,"meta":{},"queue":"default","source":"oban","system_time":1716836940646318048,"tags":[],"worker":"Asciinema.Streaming.GC"}

19:09:00.654 [info] {"args":{},"attempt":1,"duration":3899,"event":"job:stop","id":25954,"max_attempts":20,"meta":{},"queue":"default","queue_time":23603,"source":"oban","state":"success","tags":[],"worker":"Asciinema.Streaming.GC"}

[ku1ik]

So there's no corresponding Sent ... for the POST /login request, meaning, the request handler process never finishes. Weird there's no error though, given Cloudflare doesn't get a successful response from the server... :thinking:

[c3pmark]

Hey! Sorry to randomly jump in, but it looks like the discussion here is mixing up a couple of different SMTP security options. SMTP has two ways of enabling encryption (implicit and explicit) and three different ports (25, 465, 587).

"Implicit" means the client connects to the server with encryption already turned on. This is what the "ssl" option for gen_smtp does. Port 465 is specifically for implicit encryption, so if you use 465, you have to have "ssl" turned on and "tls" turned off. No other port will work with "ssl" turned on.

"Explicit" means the client connects to the server with no encryption, then uses the STARTTLS command to convert the connection to an encrypted one. This is what the "tls" option for gen_smtp does. For this you can use port 25 or 587, but port 465 will not work.

Your best bet is to stick to explicit encryption using:

SMTP_PORT=587
SMTP_TLS=always

[ku1ik]

Thanks Mark. I thought it's something like this but couldn't put my finger on it. This explanation makes it super clear.

Given the above, I think we should either rename the issue to mention "SSL + port 465" (if what you guys want here is implicit encryption on port 465), or close it (if you want explicit encryption on port 465, which is impossible).