Feature Suggestion: Disabling Domains and Accounts

[RobSlgm]

Description

I'm thrilled to share that I recently migrated some email domains from a Postfix/Dovecot setup to Mox. It's been a pleasure using this modern solution over the past few weeks. While making the switch, I gained some insights that I'd like to propose as potential improvements.

Currently, when migrating domains between email servers using Mox, there's a challenge:

• Mox immediately activates a newly created domain, acquiring certificates and allowing incoming and outgoing emails for any associated accounts/mailboxes.
• This behavior can complicate migrations where you need to import emails from the previous server before activating the new domain.

Proposed Solution: Disabling Domains and Accounts

This feature introduces two flags for disabling domains and accounts during migration:

1. Disable Flag for Mailboxes/Accounts

• When set, Mox will:
  • Reject incoming emails from external senders.
    • Disallow sending emails from the associated mailboxes.
    • Mimic the behavior of a non-existent email address.
• Important: This flag should NOT affect: - Importing/exporting emails. - IMAP access (to check migration with a mail client; or using tools such imapsync). - Access to the admin pages or mox commands

2. Disable Flag for Domains

• When set, Mox will: - Configure the domain with necessary keys (like current behavior). - NOT establish or allow external connections.
• This effectively disables all newly created accounts/mailboxes within the domain, inheriting the behavior of the "disable flag for mailboxes" described above.
• Purpose: - Allows setting up the correct DNS entries (DKIM, etc.) and preparing the DNS configuration before activating the domain. - Bulk data migration

Benefits

This feature simplifies domain migrations by:

• Preventing unintended email activity during the migration process.
• Facilitating DNS configuration by allowing key generation without establishing external connections.
• Enabling proper verification of the migration setup before activating the new domain.

[mjl-]

Thanks @RobSlgm, functionality to help with migrations makes sense.

Indeed, preventing incoming/outgoing SMTP, while allowing access to messages with IMAP (for copying data) or direct import/export seems useful.

For SMTP, we would reject outgoing messages from disabled accounts/domains during submission. For incoming messages, I'm not sure if it is all that great to have the default behaviour: it would cause messages to be permanently rejected, i.e. senders would get a bounce and not retry again. If messages are coming in, presumably the switch has been made and messages should be accepted. If messages should be rejected, perhaps a temporary error code would be better, so senders will retry up to a day (can be the time needed to get the setup ready). For disabled domains, we wouldn't send any outgoing TLS/DMARC reports.

Perhaps it's enough to only mark domains as disabled? All addresses for that domain at any account would automatically be treated as disabled. An account could have an address at an already-migrated domain (that accepts incoming and outgoing deliveries), and another address that is still disabled. Or are there advantages to marking only some address/accounts of a domain as disabled? Did you migrate accounts one at a time? I would think you'ld enable the domain about when you update the MX record.

We may also want to provide more help during the DNS migration. We currently provide a (pretty big) list of DNS records to set. Easy for new domains, but domains that will be migrated will already have records. We may want to provide some guidance on the order in which to make changes, also keeping DNS TTLs and possibly any MTA-STS policy lifetime.

What do migrations commonly look like? I think my migration went like this:

• Reduce DNS TTLs for relevant DNS records.
• Set up new machine with new mox instance.
• Add/update most DNS records for domain.
• Create accounts.
• Switch over MX record. Should be picked up quickly with short TTLs.
• Shut down old SMTP server.
• New messages now start coming in at new machine.
• Copy messages from old to new machine. Import messages into accounts.
• More DNS changes (e.g. request DMARC reports).

Perhaps there are good IMAP syncing tools to migrate messages. If it has full sync support, it can incrementally copy over any messages that are still arriving at the old SMTP server.

DNS-related changes to make for a migration:

Preparation:

• Reduce TTL and waiting for old DNS records with longer TTLs to expire.
• Add host-related (as opposed to domain-related) DNS records (DANE, host-SPF, host-TLS-reporting).
• Already add new IPs to domain SPF record.
• Already add new DKIM keys for domain (independent of existing records).
• Update existing MTA-STS policy, adding a possible new MX host to the allowlist. If there is none yet, it may be better to way until after migration.

Switch:

• MX record.

After:

• Enable DMARC and TLS reporting (request incoming reports).
• Autoconfig/autodiscover/SRV (for config discovery; if the smtp/imap hostname must stay the same, this could complicate the migration depending on existing configuration: updating a CNAME like mail.<domain> will be easy, but if the configured connections reference the old/previous hostname directly, it would complicate the required changes).
• CAA for reducing ACME CAs or account binding.
• Increase DNS TTLs and MTA-STS policy lifetime.

Curious if you had the same or different approach. How did you verify the domain config at the new mox machine was correct? I can imagine you would want to briefly activate a domain to see if incoming messages will be accepted properly, without updating the MX record yet.

Anyway, suggestions for DNS changes could just be documentation. At a later stage, it would be nice to query the existing DNS config, suggest the specific changes to make, and possibly even making them (with a click of the button). But we need DNS updating capabilities first, that's still a way off.

For now, just marking a domain as disabled for sending/receiving email seems like a good first step.

[RobSlgm]

For SMTP, we would reject outgoing messages from disabled accounts/domains during submission. For incoming messages, I'm not sure if it is all that great to have the default behaviour: it would cause messages to be permanently rejected, i.e. senders would get a bounce and not retry again. If messages are coming in, presumably the switch has been made and messages should be accepted. If messages should be rejected, perhaps a temporary error code would be better, so senders will retry up to a day (can be the time needed to get the setup ready). For disabled domains, we wouldn't send any outgoing TLS/DMARC reports.

Agree. temporary error is the right way for incoming mail.

Disabled domain should also not request ACME certificates, the ACME challenge should be done after activation.

Perhaps it's enough to only mark domains as disabled? All addresses for that domain at any account would automatically be treated as disabled. An account could have an address at an already-migrated domain (that accepts incoming and outgoing deliveries), and another address that is still disabled. Or are there advantages to marking only some address/accounts of a domain as disabled? Did you migrate accounts one at a time? I would think you'ld enable the domain about when you update the MX record.

Yes, as described it makes more sense to disable domains only. No, because I was confused about an other issue:

The big obstacle wasn't the DNS switch-over, but supporting the users. My old and new server setup has a main domain for the mail server while all the mailboxes are related to other domains. I choose a new main domain for Mox to simplify the migration as I found no working scenario for an in-place migration (i.e. using the same main domain).

• Some users had the main domain in the mail client setting...
• But most difficult and time-consuming was setting the password for each user. The old system stores "only" password hashes. (I will formulate later a further feature suggestion for an Admin API)

After setting password was done individually for each user I was wishing to be able to disable the IMAP access on mailbox level on the old system. Some users had still access on a different mail client to the old mailbox. So the suggestion should properly be: Disable mailboxes or accounts for IMAP access.

What do migrations commonly look like? I think my migration went like this:

• Reduce DNS TTLs for relevant DNS records.
• Set up new machine with new mox instance.
• Add/update most DNS records for domain.
• Create accounts.
• Switch over MX record. Should be picked up quickly with short TTLs.
• Shut down old SMTP server.
• New messages now start coming in at new machine.
• Copy messages from old to new machine. Import messages into accounts.
• More DNS changes (e.g. request DMARC reports).

• Setup a new machine with a new mox instance
• Add a test domain for mail accounts
• Check with test domain and some tools if server is working correctly and communication with external parties works
• After about 1-2 weeks start to migrate some domains from the old server to the new server
  • Disable domain on old server - no new messages are received on old server
  • Create domain on new server with accounts and aliases
  • Migrate data (mox import) which has been rsync from the old server
  • Depending on the time the data migrations takes, switch DNS MX to new server. For verifying if the migration is complete newly incoming mails are difficult to handle.
  • Enable domain
  • Support the mail client changes of users
• Keep the old server running at least till the last domain is disabled and migrated

Perhaps there are good IMAP syncing tools to migrate messages. If it has full sync support, it can incrementally copy over any messages that are still arriving at the old SMTP server.

A tool named imapsync does incremental copy very well, but unfortunately it works on single mailboxes only and takes (at least in my use-case) too long. It further requires the password of each mailbox which IMHO makes it unsuitable for a full domain migration.

The current mox import is quite fast and doesn't require user passwords. It works on single folders, which had the side benefit that some legacy naming of folders (e.g. mail clients who created standard folders using the local language) could be fixed by the migration script. It also allowed to merge mailboxes. mox import doesn't handle incremental import (no duplicate checks) so the timing of the switch over between old and new server was critical.

The migration of alias-definition requires with the current state of implementation some workarounds.

• Some issues are related to the fact that the migration was done domain after domain. So the sequence of domain migration was important, as Mox doesn't support external addresses as target of an alias.
• Resolve alias to alias chains, i.e: Alias [email protected] -> Alias [email protected] and Alias [email protected] -> Mailbox [email protected]. This because Mox doesn't support Aliases as target of an alias on the same mail server.

Curious if you had the same or different approach. How did you verify the domain config at the new mox machine was correct? I can imagine you would want to briefly activate a domain to see if incoming messages will be accepted properly, without updating the MX record yet.

See above. New server with some small domain to verify the setup and perform tests. Scripting - if and where possible - for the other domains. It'd helped that the old server had no DANE, MTS_STA or auto-discovery setup, so much fewer conflicting DNS records.

For now, just marking a domain as disabled for sending/receiving email seems like a good first step.

Thanks!

[DanielG]

Hi Robert,

On Mon, Jun 10, 2024 at 11:47:27PM -0700, Robert Bill wrote:

Perhaps there are good IMAP syncing tools to migrate messages. If it has full sync support, it can incrementally copy over any messages that are still arriving at the old SMTP server.

A tool named imapsync does incremental copy very well, but unfortunately it works on single mailboxes only and takes (at least in my use-case) too long.

FYI I use https://guilhem.org/interimap/ for offline mail, but it also works very well for keeping my mox test mailbox in-sync with my old mail setup while I'm still evaluating.

It further requires the password of each mailbox which IMHO makes it unsuitable for a full domain migration.

With interimap you can have it ssh onto the mail server and/or launch a dovecot imap process directly, so no passwords on the old system are necessary with proper permission setup. On the mox side it's still necessary right now, but I would expect it should be possible to support a doveadm exec imap style imap connection in mox too if Michiel agrees this is useful.

That would solve the migration/sync problem in my mind (NOTIFY support would be nice-to-have still ;)

--Daniel

[naddika]

I'm about to migrate my Postfix emails onto Mox as well. I've made a backup of /var/vmail directory, which is Maildir, wherein my emails are stored. Do I understand correctly that simply restoring, or uploading, them in Mox, via UI, be enough for to perform migration?

[RobSlgm]

I'm about to migrate my Postfix emails onto Mox as well. I've made a backup of /var/vmail directory, which is Maildir, wherein my emails are stored. Do I understand correctly that simply restoring, or uploading, them in Mox, via UI, be enough for to perform migration?

@naddika That's my understanding, but not what I did and referred above. I used a bash script iterating the /var/vmail directory to generate mox import statements for each relevant folder.

./mox import maildir somebody Inbox /var/vmail/example,com/somebody/
./mox import maildir somebody Junk /var/vmail/example,com/somebody/.Junk/
./mox import maildir somebody Drafts /var/vmail/example,com/somebody/.Entw\&APw-rfe/

So re-directions (see last line) are possible and even merging folders, e.g ./mox import maildir somebody Inbox /var/vmail/example.org/otheraccount/ .

The creation of account and addresses (and also alias) can also be scripted

./mox config account add somebody [email protected]
./mox config address add [email protected] somebody

This basically allows you to test and re-run the whole import.

[naddika]

Thanks

[mjl-]

@RobSlgm:

I started working on a "disabled" seting for domains. It will reject incoming/outgoing messages involving a disabled domain (during smtp, but also when sending through the web interfaces). And it will prevent fetching TLS certs with ACME.

But most difficult and time-consuming was setting the password for each user. The old system stores "only" password hashes.

How did you handle this? First create new accounts on the new machine and ask users to login and set their (new) password? Or did you want to way to move the passwords (or hashes) from the old system? I can imagine authentication middleware that has access to incoming plain passwords, allowing you to set them in the new mail server. Wouldn't work for users don't log in for a while, or when SCRAM is used.

Disable mailboxes or accounts for IMAP access.

Yes, this makes sense. I think we can add a "login disabled" field to each account, with a configurable message to show to users when they try to login in. So could be shown something like: "login disabled: your account has been migrated, https://example.org/migration for instructions". I think the login-disabled setting for an account should prevent all user-initiated login attempts. Also into the webmail/account web interfaces and smtp. Only the admin can still do operations on the account, like exporting files. And internal access to the account would still work, including incoming deliveries. I think this will be useful in general, also outside of migrations.

@DanielG: I like your suggestion for a "doveadm exec imap"-like command for mox. Didn't know that existed, and that tools are making use of it. I think I can relatively easily add a "mox admin imapserve user@domain" command (and similar "smtpserve") that serves a preauthenticated imap connection on fd 0. It will go through the ctl file (admins only), and will also work when an account is disabled (so regular user access is disabled, but admins can use this approach for syncing).

[mjl-]

The first commit above add the ability to mark domains as disabled, and allows accounts to be disabled for logging in. See details in the commit message. The second commit adds "mox admin imapserve $address" that opens up an imap session preauthenticated as the user. I tested it with mutt, not interimap.

Thanks again for reporting, more feedback welcome!