mox Another go with fly.io

Another go with fly.io

Open delano opened this issue 7 months ago • 3 comments

I've been playing around with the idea of running a mail server on fly.io for a while and had some time this weekend to look into it. Similar to your sentiment on #14, it's been a good reason to learn more about fly.io at the same time. The same issues with sending mail that were mentioned in #14 still apply, but I was curious to see if there were any updates or new learnings since then, so I decided to give it a shot and document my findings here.

There's something really cool about a mail server that can spin down to 0 machines and quickly re-hydrate it self when a message comes in. It's an interesting twist on how we might get to a place where we (as individuals) run more of our own services again.

Steps

Anyway, here's what I did:

Modifed the Dockerfile in main
- To use the prebuilt image instead of building mox (this was just to simplify but the Dockerfile as-is should work fine too).
Ran docker build -t mox .
- This built the image locally so I could run the mox quickstart command to generate the initial configuration. This was a notable shortcut that avoided dealing with fly volumes but has its obvious drawbacks: (1) the IP detection and other heuristics obviously don't match the fly.io environment and (2) the generated passwords, keys, and certs end up in the deployed image which is not ideal for security reasons (although I have a solution for that which I'll mention at the end).
Created an initial fly.toml.
- This Fly.io community thread about receiving email / inbound smtp from 2021 was helpful in getting started. (They're using Haraka and the person that started the thread mentioned they are (or were) running it on fly.io for noreply.zone). I say initial b/c running the launch command modifies it in place.
Ran fly launch.
- Answered y to copying the existing config.
- Answered y to adding public 1pv4 and ipv6 addresses.
Updated the Listeners config in mox.conf
- Left internal IPs as 127.0.0.1
- Set public IPs to 0.0.0.0 and ::0.
- Set public NATIPs to the fly ipv4 and ipv6 addresses (since they are only available after lanuch -- same as #14).
Ran fly deploy to re-reploy the app with the updated config.

This got things running but I ran into a few issues:

I'd a hard time getting the fly proxy HTTP/service checks to pass. I'm going to open a thread on this in the Fly forum as well, but part of the challenge was understanding how mox services were meant to run on 127.0.0.1 vs public IPs (and NATs). Not a critisism of either, just a lot of complexity that I haven't got to the bottom of yet.
Receiving mail on ports 465 and 587 worked, but not on port 25 (smtp). Probably related to the first point (e.g. the failing checks affect how/if the fly proxy routes traffic).
Sending mail I didn't check. It wasn't the focus and even

tl;dr

$ docker build -t mox .
$ docker run --rm -v $(pwd)/mox_data:/home/mox mox-mta /bin/mox quickstart -skipdial -hostname mail.example.com [email protected] mox
$ fly launch [-o orgname -r region]
$ fly ips list
$ fly status

# After manually updating the fly.toml, mox.conf
$ fly deploy
$ fly ssh console -s

Future improvements

Using fly certs to "prime" the TLS before the web services start. It wouldn't replace it since you want SMTP to be able to negotiate with its own certs but it would be nice to have a more robust TLS story.
Using fly launch args --file-literal, --file-local, and --file-secret to inject config and data into the container at build time. This
Looking into the fly.toml [deploy] setting called release_command. e.g. for running release_command = "mox quickstart".
- Note that volumes are not available at build time so the files would still be written to the ephemeral storage.
Sorting out the aforementioned issues with the Fly proxy service checks.

Files

`fly.toml`

Details

app = 'mox'
primary_region = 'ams'

[build]
  dockerfile = 'Dockerfile'

[deploy]
  wait_timeout = '3m0s'

[http_service]
  internal_port = 80
  force_https = false
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[services]]
  protocol = 'tcp'
  internal_port = 25

  [[services.ports]]
    port = 25

  [[services.ports]]
    port = 465
    handlers = ['tls']

  [[services.ports]]
    port = 587

  [[services.ports]]
    port = 80
    handlers = ['http']

  [[services.ports]]
    port = 443
    handlers = ['http', 'tls']

  [[services.ports]]
    port = 993
    handlers = ['tls']

  [[services.ports]]
    port = 143

  [[services.ports]]
    port = 8010

[[restart]]
  policy = 'never'
  retries = 1
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

`Dockerfile`

Details

ARG GO_VERSION=1
FROM golang:${GO_VERSION}-bookworm as builder

# Create mox user and homedir (or pick another name or homedir):
RUN useradd -m -d /home/mox mox

WORKDIR /home/mox

# Download and install mox:
RUN apt-get update && \
    apt-get install -y curl ca-certificates gnupg2 tzdata vim \
                       iputils-ping dnsutils iproute2 nmap netcat-openbsd

# Find the latest release: https://beta.gobuilds.org/github.com/mjl-/mox@latest/linux-amd64-latest/
RUN curl -v -O -L https://beta.gobuilds.org/github.com/mjl-/[email protected]/linux-amd64-go1.22.5/0pAbuhqp5CMBgh73mTHluWLML6-I/mox-v0.0.11-go1.22.5.gz

RUN gunzip mox-v0.0.11-go1.22.5.gz
RUN mv mox-v0.0.11-go1.22.5 mox
RUN chmod +x mox
RUN mv mox /bin/

# When setting up a new system, don't run the mox quickstart in this
# Dockerfile. Instead we build and then run the container with a volume
# for the config and data, overriding the CMD. The config files generated
# are then made for linux (so they include the mox.service) and then
# when the image runs the first time all those files are inplace.

COPY mox_data /home/mox
RUN chown -R 1000:0 /home/mox/config
RUN chown -R 1000:0 /home/mox/data

# SMTP for incoming message delivery.
EXPOSE 25/tcp

# SMTP/submission with TLS.
EXPOSE 465/tcp

# SMTP/submission without initial TLS.
EXPOSE 587/tcp

# HTTP for internal account and admin pages.
EXPOSE 80/tcp

# HTTPS for ACME (Let's Encrypt), MTA-STS and autoconfig.
EXPOSE 443/tcp

# IMAP with TLS.
EXPOSE 993/tcp

# IMAP without initial TLS.
EXPOSE 143/tcp

# Prometheus metrics.
EXPOSE 8010/tcp

#CMD pwd && ls -lha
CMD ["/bin/mox", "serve"]

`mox.conf`

Details

# NOTE: This config file is in 'sconf' format. Indent with tabs. Comments must be
# on their own line, they don't end a line. Do not escape or quote strings.
# Details: https://pkg.go.dev/github.com/mjl-/sconf.

# https://www.xmox.nl/config/

# Directory where all data is stored, e.g. queue, accounts and messages, ACME TLS
# certs/keys. If this is a relative path, it is relative to the directory of
# mox.conf.
DataDir: ../data

# Default log level, one of: error, info, debug, trace, traceauth, tracedata.
# Trace logs SMTP and IMAP protocol transcripts, with traceauth also messages with
# passwords, and tracedata on top of that also the full data exchanges (full
# messages), which can be a large amount of data.
LogLevel: debug

# User to switch to after binding to all sockets as root. Default: mox. If the
# value is not a known user, it is parsed as integer and used as uid and gid.
# (optional)
User: mox

# Full hostname of system, e.g. mail.<domain>
Hostname: mail.example.com

# If enabled, a single DNS TXT lookup of _updates.xmox.nl is done every 24h to
# check for a new release. Each time a new release is found, a changelog is
# fetched from https://updates.xmox.nl/changelog and delivered to the postmaster
# mailbox. (optional)
#
# RECOMMENDED: please enable to stay up to date
#
#CheckUpdates: true

# Automatic TLS configuration with ACME, e.g. through Let's Encrypt. The key is a
# name referenced in TLS configs, e.g. letsencrypt. (optional)
ACME:
	letsencrypt:

		# For letsencrypt, use https://acme-v02.api.letsencrypt.org/directory.
		DirectoryURL: https://acme-v02.api.letsencrypt.org/directory

		# Email address to register at ACME provider. The provider can email you when
		# certificates are about to expire. If you configure an address for which email is
		# delivered by this server, keep in mind that TLS misconfigurations could result
		# in such notification emails not arriving.
		ContactEmail: [email protected]

		# If set, used for suggested CAA DNS records, for restricting TLS certificate
		# issuance to a Certificate Authority. If empty and DirectyURL is for Let's
		# Encrypt, this value is set automatically to letsencrypt.org. (optional)
		IssuerDomainName: letsencrypt.org

# File containing hash of admin password, for authentication in the web admin
# pages (if enabled). (optional)
AdminPasswordFile: adminpasswd

# Listeners are groups of IP addresses and services enabled on those IP addresses,
# such as SMTP/IMAP or internal endpoints for administration or Prometheus
# metrics. All listeners with SMTP/IMAP services enabled will serve all configured
# domains. If the listener is named 'public', it will get a few helpful additional
# configuration checks, for acme automatic tls certificates and monitoring of ips
# in dnsbls if those are configured.
Listeners:
	internal:

		# Use 0.0.0.0 to listen on all IPv4 and/or :: to listen on all IPv6 addresses, but
		# it is better to explicitly specify the IPs you want to use for email, as mox
		# will make sure outgoing connections will only be made from one of those IPs. If
		# both outgoing IPv4 and IPv6 connectivity is possible, and only one family has
		# explicitly configured addresses, both address families are still used for
		# outgoing connections. Use the "direct" transport to limit address families for
		# outgoing connections.
		IPs:
			- 127.0.0.1

		# If empty, the config global Hostname is used. (optional)
		Hostname: mail.example.com

		# Account web interface, for email users wanting to change their accounts, e.g.
		# set new password, set new delivery rulesets. Default path is /. (optional)
		AccountHTTP:
			Enabled: true

		# Admin web interface, for managing domains, accounts, etc. Default path is
		# /admin/. Preferably only enable on non-public IPs. Hint: use 'ssh -L
		# 8080:localhost:80 you@yourmachine' and open http://localhost:8080/admin/, or set
		# up a tunnel (e.g. WireGuard) and add its IP to the mox 'internal' listener.
		# (optional)
		AdminHTTP:
			Enabled: true

		# Webmail client, for reading email. Default path is /webmail/. (optional)
		WebmailHTTP:
			Enabled: true

		# Like WebAPIHTTP, but with plain HTTP, without TLS. (optional)
		WebAPIHTTP:
			Enabled: true

		# All configured WebHandlers will serve on an enabled listener. (optional)
		WebserverHTTP:
			Enabled: false

		# Serve prometheus metrics, for monitoring. You should not enable this on a public
		# IP. (optional)
		MetricsHTTP:
			Enabled: false

	public:

		# Use 0.0.0.0 to listen on all IPv4 and/or :: to listen on all IPv6 addresses, but
		# it is better to explicitly specify the IPs you want to use for email, as mox
		# will make sure outgoing connections will only be made from one of those IPs. If
		# both outgoing IPv4 and IPv6 connectivity is possible, and only one family has
		# explicitly configured addresses, both address families are still used for
		# outgoing connections. Use the "direct" transport to limit address families for
		# outgoing connections.
		IPs:
			- 0.0.0.0
			- ::

		# If set, the mail server is configured behind a NAT and field IPs are internal
		# instead of the public IPs, while NATIPs lists the public IPs. Used during
		# IP-related DNS self-checks, such as for iprev, mx, spf, autoconfig,
		# autodiscover, and for autotls. (optional)
		NATIPs:
			- 1.12.123.234
			- 0a01:01ff:2::5a:e11b:0

		# For SMTP/IMAP STARTTLS, direct TLS and HTTPS connections. (optional)
		TLS:

			# Name of provider from top-level configuration to use for ACME, e.g. letsencrypt.
			# (optional)
			ACME: letsencrypt

			# Private keys used for ACME certificates. Specified explicitly so DANE TLSA DNS
			# records can be generated, even before the certificates are requested. DANE is a
			# mechanism to authenticate remote TLS certificates based on a public key or
			# certificate specified in DNS, protected with DNSSEC. DANE is opportunistic and
			# attempted when delivering SMTP with STARTTLS. The private key files must be in
			# PEM format. PKCS8 is recommended, but PKCS1 and EC private keys are recognized
			# as well. Only RSA 2048 bit and ECDSA P-256 keys are currently used. The first of
			# each is used when requesting new certificates through ACME. (optional)
			HostPrivateKeyFiles:
				- hostkeys/mail.example.com.20240706T223117.rsa2048.privatekey.pkcs8.pem
				- hostkeys/mail.example.com.20240706T223117.ecdsap256.privatekey.pkcs8.pem

		# (optional)
		SMTP:
			Enabled: true

			# Addresses of DNS block lists for incoming messages. Block lists are only
			# consulted for connections/messages without enough reputation to make an
			# accept/reject decision. This prevents sending IPs of all communications to the
			# block list provider. If any of the listed DNSBLs contains a requested IP
			# address, the message is rejected as spam. The DNSBLs are checked for healthiness
			# before use, at most once per 4 hours. IPs we can send from are periodically
			# checked for being in the configured DNSBLs. See MonitorDNSBLs in domains.conf to
			# only monitor IPs we send from, without using those DNSBLs for incoming messages.
			# Example DNSBLs: sbl.spamhaus.org, bl.spamcop.net. See
			# https://www.spamhaus.org/sbl/ and https://www.spamcop.net/ for more information
			# and terms of use. (optional)
			#DNSBLs:
				#- sbl.spamhaus.org
				#- bl.spamcop.net

		# SMTP over TLS for submitting email, by email applications. Requires a TLS
		# config. (optional)
		Submissions:
			Enabled: true

		# IMAP over TLS for reading email, by email applications. Requires a TLS config.
		# (optional)
		IMAPS:
			Enabled: true

		# Serve autoconfiguration/autodiscovery to simplify configuring email
		# applications, will use port 443. Requires a TLS config. (optional)
		AutoconfigHTTPS:
			Enabled: true

		# Serve MTA-STS policies describing SMTP TLS requirements. Requires a TLS config.
		# (optional)
		MTASTSHTTPS:
			Enabled: true

		# All configured WebHandlers will serve on an enabled listener. Either ACME must
		# be configured, or for each WebHandler domain a TLS certificate must be
		# configured. (optional)
		WebserverHTTPS:
			Enabled: true

# Destination for emails delivered to postmaster addresses: a plain 'postmaster'
# without domain, 'postmaster@<hostname>' (also for each listener with SMTP
# enabled), and as fallback for each domain without explicitly configured
# postmaster destination.
Postmaster:
	Account: d

	# E.g. Postmaster or Inbox.
	Mailbox: Postmaster

# Destination for per-host TLS reports (TLSRPT). TLS reports can be per recipient
# domain (for MTA-STS), or per MX host (for DANE). The per-domain TLS reporting
# configuration is in domains.conf. This is the TLS reporting configuration for
# this host. If absent, no host-based TLSRPT address is configured, and no host
# TLSRPT DNS record is suggested. (optional)
HostTLSRPT:

	# Account to deliver TLS reports to. Typically same account as for postmaster.
	Account: d

	# Mailbox to deliver TLS reports to. Recommended value: TLSRPT.
	Mailbox: TLSRPT

	# Localpart at hostname to accept TLS reports at. Recommended value: tls-reports.
	Localpart: tls-reports

`.gitignore`

Details

This kept all of the most sensitive files out of the git repo, and also kept things like mox binary from being committed.

mox
**/mox
adminpasswd
accounts/
acme/
queue/
*.key
*.crt
*.db

Jul 08 '24 02:07 delano

mox mox copied to clipboard

Another go with fly.io

Steps

tl;dr

Future improvements

Files

fly.toml

Dockerfile

mox.conf

.gitignore

mox
mox copied to clipboard

`fly.toml`

`Dockerfile`

`mox.conf`

`.gitignore`