mailcow-dockerized icon indicating copy to clipboard operation
mailcow-dockerized copied to clipboard

DB errors on Cold Standby system

Open apharp opened this issue 2 years ago • 0 comments

Contribution guidelines

I've found a bug and checked that ...

  • [x] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
  • [X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
  • [X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
  • [X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

If you attempt to use the create_cold_stanby script from a non-encrypted origin host to a LUKS encrypted standby system, you will not be able to start mailcow on the standby system.

If happens only in the very specific situation where copying from a non-encrypted system to an encrypted system. I have been able to reproduce this across multiple system on fresh Mailcow installs.

Logs

mailcowdockerized-mysql-mailcow-1      | 2022-08-05 15:53:07 0 [Note] Server socket created on IP: '::'.
mailcowdockerized-mysql-mailcow-1      | 2022-08-05 15:53:07 0 [ERROR] mysqld: Can't create/write to file '/run/mysqld/mysqld.pid' (Errcode: 13 "Permission denied")
mailcowdockerized-mysql-mailcow-1      | 2022-08-05 15:53:07 0 [ERROR] Can't start server: can't create PID file: Permission denied

Steps to reproduce

System 1 = unencrypted drive System 2 = LUKS encrypted drive

  1. Run create_cold_standby.sh on system1 cloning to system2.
  2. SSH into system2 after clone completes and attempt to start mailcow.

System information

Question Answer
My operating system Debian 11
Is Apparmor, SELinux or similar active? No
Virtualization technology (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported KVM
Server/VM specifications (Memory, CPU Cores) 8GB RAM / 8 CPU Cores
Docker version (docker version) 20.10.17, build 100c701
docker-compose version (docker-compose version) v2.9.0
mailcow version (git describe --tags `git rev-list --tags --max-count=1` ) 2022-07a-1-g67c04052
Reverse proxy (custom solution) None

Output of git diff origin/master, any other changes to the code? If so, please post them:

diff --git a/create_cold_standby.sh b/create_cold_standby.sh
index 924339af..24623b4b 100755
--- a/create_cold_standby.sh
+++ b/create_cold_standby.sh
@@ -1,7 +1,7 @@
 #!/bin/bash

-export REMOTE_SSH_KEY=/root/.ssh/id_rsa
-export REMOTE_SSH_PORT=22
-export REMOTE_SSH_HOST=my.remote.host
+export REMOTE_SSH_KEY=/root/.ssh/id_ed25519
+export REMOTE_SSH_PORT=11381
+export REMOTE_SSH_HOST=<<REDACTED>>

 /opt/mailcow-dockerized/helper-scripts/_cold-standby.sh

All third-party firewalls and custom iptables rules are unsupported. Please check the Docker docs about how to use Docker with your own ruleset. Nevertheless, iptabels output can help us to help you: iptables -L -vn:

# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  306 49155 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  306 49155 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
  306 49155 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
   50  3996 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
  306 49155 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

ip6tables -L -vn:

# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1194 92304 DOCKER-USER  all      *      *       ::/0                 ::/0
 1194 92304 DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0
 1194 92304 DOCKER     all      *      br-mailcow  ::/0                 ::/0
    6   360 ACCEPT     all      *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all      br-mailcow !br-mailcow  ::/0                 ::/0
 1188 91944 ACCEPT     all      br-mailcow br-mailcow  ::/0                 ::/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:110
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:143
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:4190
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:993
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:995
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::f  tcp dpt:443
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::f  tcp dpt:80
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:465
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:587
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:25

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all      br-mailcow !br-mailcow  ::/0                 ::/0
 1194 92304 RETURN     all      *      *       ::/0                 ::/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all      *      br-mailcow  ::/0                 ::/0
    0     0 RETURN     all      *      *       ::/0                 ::/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 1194 92304 RETURN     all      *      *       ::/0                 ::/0

iptables -L -vn -t nat:

# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  311 14546 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

ip6tables -L -vn -t nat:

# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all      *      br-mailcow  ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL
    0     0 MASQUERADE  all      *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:143
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:443
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:80
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:25

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all      br-mailcow *       ::/0                 ::/0
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::e]:110
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::e]:143
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::e]:4190
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::e]:993
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::e]:995
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::f]:443
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::f]:80
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::d]:465
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::d]:587
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::d]:25

DNS problems? Please run docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 (set the IP accordingly, if you changed the internal mailcow network) and post the output:

YOUR OUTPUT GOES HERE

apharp avatar Aug 05 '22 20:08 apharp