mailcow-dockerized icon indicating copy to clipboard operation
mailcow-dockerized copied to clipboard

Backup Script failed

Open t-lie opened this issue 1 year ago โ€ข 9 comments

Contribution guidelines

I've found a bug and checked that ...

  • [X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
  • [X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
  • [X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
  • [X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

I am using mailcow-dockerized on pysical debian machine.
Since today the backup script failed with this error:

Found project name mailcowdockerized
/bin/tar: /vmail: Cannot stat: No such file or directory
/bin/tar: Exiting with failure status due to previous errors

mailcow is running wihtout any issue. there was no changes in config.

Logs:

which logs are useful here? docker compose logs shows many entries, what should not be useful here.

Steps to reproduce:

BU Script was started with ./helper-scripts/backup_and_restore.sh backup vmail

Which branch are you using?

master

Operating System:

debian 11.5

Server/VM specifications:

16GB RAM, 4 CPU

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

no

Docker version:

20.10.18

docker-compose version or docker compose version:

v2.10.2

mailcow version:

2022-11

Reverse proxy:

no

Logs of git diff:

nothing

Logs of iptables -L -vn:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 207K   61M DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 207K   61M DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 2715  537K ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   61  3660 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
 2517 6105K ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
 150K   44M ACCEPT     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
12569  778K DOCKER     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
38886 9335K ACCEPT     all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
12321  764K ACCEPT     all  --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   61  3660 ACCEPT     tcp  --  !docker0 docker0  0.0.0.0/0            172.17.0.2           tcp dpt:9001
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.4           tcp dpt:3306
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:443
    4   256 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:80
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:587
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.6           tcp dpt:8983
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:465
   45  2372 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:25
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993
  199 11908 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 2517 6105K DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
38886 9335K DOCKER-ISOLATION-STAGE-2  all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
 207K   61M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
41403   15M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 207K   61M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Logs of ip6tables -L -vn:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
35878   45M DOCKER-USER  all      *      *       ::/0                 ::/0                
35878   45M DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0                
35878   45M DOCKER     all      *      br-mailcow  ::/0                 ::/0                
28479   44M ACCEPT     all      *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all      br-mailcow !br-mailcow  ::/0                 ::/0                
 7399  501K ACCEPT     all      br-mailcow br-mailcow  ::/0                 ::/0                

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::5  tcp dpt:443
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::5  tcp dpt:80
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:143
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:4190
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:993
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:995
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:110
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::6  tcp dpt:465
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::6  tcp dpt:587
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::6  tcp dpt:25

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all      br-mailcow !br-mailcow  ::/0                 ::/0                
35878   45M RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all      *      br-mailcow  ::/0                 ::/0                
    0     0 RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
35878   45M RETURN     all      *      *       ::/0                 ::/0

Logs of iptables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1266  123K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   12   720 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
 8823  669K MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0           
    0     0 MASQUERADE  tcp  --  *      *       172.17.0.2           172.17.0.2           tcp dpt:9001
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.4           172.22.1.4           tcp dpt:3306
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.6           172.22.1.6           tcp dpt:8983
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  br-mailcow *       0.0.0.0/0            0.0.0.0/0           
   61  3660 DNAT       tcp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9001 to:172.17.0.2:9001
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.4:3306
   12   720 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.5:443
    4   256 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.5:80
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.6:8983
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465
   45  2372 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993
  199 11908 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110

Logs of ip6tables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all      *      br-mailcow  ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL
    0     0 MASQUERADE  all      *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0                
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::5  fd4d:6169:6c63:6f77::5  tcp dpt:443
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::5  fd4d:6169:6c63:6f77::5  tcp dpt:80
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:143
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::6  fd4d:6169:6c63:6f77::6  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::6  fd4d:6169:6c63:6f77::6  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::6  fd4d:6169:6c63:6f77::6  tcp dpt:25

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all      br-mailcow *       ::/0                 ::/0                
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::5]:443
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::5]:80
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::10]:143
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::10]:4190
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::10]:993
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::10]:995
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::10]:110
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::6]:465
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::6]:587
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::6]:25

DNS check:

ok

t-lie avatar Dec 18 '23 10:12 t-lie

Any special config? For example: Custom Docker Volume location?

DerLinkman avatar Dec 18 '23 13:12 DerLinkman

Any special config? For example: Custom Docker Volume location?

The only special is custom docker root. Because of storage issue I have moved them from /var/lib/docker to /vol/data/dockerroot and created /etc/docker/daemon.json with following entry: { "data-root": "/vol/data/dockerroot" }

Do I need to change it anywhere in BU script?

Thanks

t-lie avatar Dec 18 '23 14:12 t-lie

I actually was not aware that this will cause any trouble for now...

Can you try the following:

docker volume ls -qf name=mailcowdockerized_vmail-vol-1

and paste the output in here?

DerLinkman avatar Dec 18 '23 14:12 DerLinkman

I actually was not aware that this will cause any trouble for now...

Can you try the following:

docker volume ls -qf name=mailcowdockerized_vmail-vol-1

and paste the output in here?

badly I get no output.

t-lie avatar Dec 18 '23 14:12 t-lie

Can you do a docker volume ls in general?

DerLinkman avatar Dec 18 '23 15:12 DerLinkman

docker volume ls

nothing... DRIVER VOLUME NAME

They are placed here:

root@deb01:/vol/data/mailcow# ls -l
insgesamt 244
drwxr-xr-x 41 _apt             systemd-journal    4096 19. Dez 06:57 clamd-db-vol-1
drwxr-xr-x  2 root             root               4096 26. Sep 2022  crypt-vol-1
drwxr-xr-x  2 systemd-coredump systemd-coredump   4096 18. Dez 09:56 mysql-socket-vol-1
drwxr-xr-x  5 systemd-coredump systemd-coredump   4096 18. Dez 09:56 mysql-vol-1
drwxr-xr-x 16 root             root               4096 26. Sep 2022  postfix-vol-1
drwxr-xr-x  2 systemd-coredump root               4096 19. Dez 08:38 redis-vol-1
drwxr-xr-x  2 systemd-network  systemd-journal  200704 19. Dez 08:40 rspamd-vol-1
drwxr-xr-x  2 systemd-coredump systemd-coredump   4096 28. Sep 2022  sogo-userdata-backup-vol-1
drwxr-xr-x 16 root             root               4096  1. Dez 2022  sogo-web-vol-1
drwxr-xr-x  5             8983             8983   4096 26. Sep 2022  solr-vol-1
drwxr-xr-x  4             5000             5000   4096 30. Mรคr 2023  vmail-index-vol-1
drwxr-xr-x  5             5000             5000   4096 26. Sep 2022  vmail-vol-1

t-lie avatar Dec 18 '23 15:12 t-lie

Can you do a docker volume ls in general?

But these volumes are placed on this external mount (/vol/data/) since installation of mailcow... so there was no change. Within docker-compose file the volumes are all set like this:

 volumes:
        - /vol/data/mailcow/mysql-vol-1:/var/lib/mysql/

Any ideas :( ?

t-lie avatar Dec 19 '23 07:12 t-lie

No one who can help here? Is there any configuration that needs to be adjusted to use custom volume and container paths in the bu script? Thanks

t-lie avatar Dec 23 '23 09:12 t-lie

new year new try... anyone can help me out here? THX

t-lie avatar Jan 03 '24 07:01 t-lie

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

milkmaker avatar Mar 03 '24 09:03 milkmaker

THIS IS A AUTOMATED MESSAGE!

It seems your issue is not a bug. Therefore we highly advise you to get support!

You can get support either by:

This issue will be closed. If you think your reported issue is not a support case feel free to comment above and if so the issue will reopened.

milkmaker avatar Mar 04 '24 11:03 milkmaker