cmdeploy run issues [Contabo]

[feld]

A user reported problems trying to deploy a server on a Debian 12 Contabo VPS and I ordered one to reproduce the problems. Here is what I found doing a clean deployment.

1. The README should tell users to set the www and mta-sts CNAMEs before the first run of cmdeploy to avoid having to hit those errors.

2. Contabo ships broken DNS servers in /etc/resolv.conf. This really messes with the deploy in several ways. I think it would be wise to immediately overwrite the system DNS recursors with known good ones (google/8.8.8.8, opendns, etc. something that just works until Unbound is running)

3. Extremely early in the cmdeploy run it needs to start checking and managing some sysctls. Contabo ships net.ipv6.conf.all.disable_ipv6=1 which prevents Unbound from working after it is installed because it cannot bind to ::1. We can just drop a file into /etc/sysctl.d/ and actually there are some important things that should be changed for hardening the TCP/IP stack that can be discussed at another time. Having a functional IPv6 stack is important regardless of whether or not we intend to use it because of these little issues.

4. opendkim-genkey fails because it is being run as user opendkim and /usr/sbin will not be in the PATH. The command should be updated to use the absolute path: /usr/sbin/opendkim-genkey

    if not host.get_fact(File, f"/etc/dkimkeys/{dkim_selector}.private"):
        server.shell(
            name="Generate OpenDKIM domain keys",
            commands=[
                f"opendkim-genkey -D /etc/dkimkeys -d {domain} -s {dkim_selector}"
            ],
            _use_su_login=True,
            _su_user="opendkim",
        )

After these changes are made the deployment work successfully.

[feld]

For the sysctls: Contabo is dropping their settings into the bottom of /etc/sysctl.d/99-sysctl.conf so an override needs to be in a file that is loaded later, like /etc/sysctl.d/9999-chatmail.conf. The service to restart is systemd-sysctl.service. I have tested it works correctly with this naming convention to override the earlier declaration.

[missytake]

1. https://github.com/chatmail/server/blob/main/cmdeploy/src/cmdeploy/cmdeploy.py#L74 should make sure that mta-sts and www CNAME's are set, and abort cmdeploy run otherwise. Any idea why that didn't work in your case?

2. we can add 9.9.9.9 to the /etc/resolv.conf in my opinion.

3. good catch.

4. isn't also a reboot necessary for enabling IPv6? Source: https://contabo.com/blog/adding-ipv6-connectivity-to-your-server/ I'd be surprised if this is possible during a pyinfra deploy. And I'm a bit wary if we should do this for every VPS on which chatmail is deployed, who knows how this could affect other setups. edit: ah, restarting systemd-sysctl is enough to avoid rebooting?

In general I think we should also advise against using contabo in the README. These errors didn't seem to come up consistently for all contabo VPSs, right? So we can catch some of them, but I'm not convinced we should clean up their mess of breaking Debian 12. Maybe we could write a short "compatibility script" for contabo users, which does these steps, and is "use at your own risk". That would be a bit more fitting for the situation imho.

[adbenitez]

In general I think we should also advise against using contabo in the README. These errors didn't seem to come up consistently for all contabo VPSs, right? So we can catch some of them, but I'm not convinced we should clean up their mess of breaking Debian 12.

if you are going to do so, also mention that Contabo limits traffic volume in port 25 and eventually you might need to try to convince support of rising limit (which I have failed so far)

[hpk42]

On Tue, Mar 11, 2025 at 05:08 -0700, adb wrote:

adbenitez left a comment (chatmail/server#515)

In general I think we should also advise against using contabo in the README. These errors didn't seem to come up consistently for all contabo VPSs, right? So we can catch some of them, but I'm not convinced we should clean up their mess of breaking Debian 12.

if you are going to do so, also mention that Contabo limits traffic volume in port 25 and eventually you might need to try to convince support of rising limit (which I have failed so far)

Maybe we can do a chatmail.at/hosting-notes.{md,html} file that has alphabetical sorted notes on various hosters? Could start with just some .md file on current repo here.