nautical-backup
nautical-backup copied to clipboard
Minituff
Skipped Containers
Hi there, I have created a simple nautical container to backup my docker containers but some containers are skipped, how can I find out why?
Thanks
I have added the debug log and found out that the container name does not matched the folder. "Normally a container is backed up only when the container-name is the exact same as the source folder name."
@alexkiddddd Thank you for the fix. I can confirm that this fixed the issue for me. In my case my container names are different from my docker data volume names, once I set the docker data volume name to the same name as the container it worked right away with require label true.
Hey @khallingstad, so all is working now? Is there anything I need to fix?
Hey @Minituff yes all working good now! In hindsight it makes total sense that docker data name needs to be the same as the container name. Thank you for a great tool!
Hi, little remark here. In the case of a docker compose file with more than one service (eg. wordpress) there would be two container names, one of which would not match the folder name... as in this case:
docker-compose.yml for nautical backup
version: '3'
services:
nautical-backup:
image: minituff/nautical-backup:latest
container_name: nautical-backup
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/user/docker:/app/source
- /home/user/dockerbackup:/app/destination
environment: # Optional variables
- LOG_LEVEL=DEBUG
- CRON_SCHEDULE=05 21 * * *
- SKIP_CONTAINERS=nautical-backup
restart: unless-stopped
and here using the new group labels:
docker-compose.yml for wordpress
version: '3.1'
services:
wordpress:
image: wordpress:6.5
container_name: wordpress-blog
restart: unless-stopped
ports:
- 80:80
environment:
WORDPRESS_DB_HOST: xxx
WORDPRESS_DB_USER: xxx
WORDPRESS_DB_PASSWORD: xxx
WORDPRESS_DB_NAME: xxx
volumes:
- ./html:/var/www/html
labels:
- "nautical-backup.group=wp"
db:
image: mysql:8.0
container_name: wordpress-db
restart: always
environment:
MYSQL_DATABASE: xxx
MYSQL_USER: uxxx
MYSQL_PASSWORD: xxx
MYSQL_RANDOM_ROOT_PASSWORD: xxx
volumes:
- ./db:/var/lib/mysql
labels:
- "nautical-backup.group=wp"
@Minituff after several attempts I can confirm the group label does not seem to work as expected (unless I have not configured it correctly, of course)
nautical backup log shows:
DEBUG: Containers completed: many here!
DEBUG: Containers skipped: wordpress-blog, wordpress-db
INFO: Success. 5 containers backed up! 2 skipped.
Hey, just taking a look at this now.
-
Are you trying to use the
nautical-backup.grouplabel to have thewordpress-blogandwordpress-dbstopped, backed up, and then started at the same time? -
Or are you just trying to back up the volumes? I see in the logs they are being skipped.
- If so, which paths would you like Nautical to backup? I only see
./htmland./db
- If so, which paths would you like Nautical to backup? I only see
Hi, thank you for your interest.
- Are you trying to use the
nautical-backup.grouplabel to have thewordpress-blogandwordpress-dbstopped, backed up, and then started at the same time?
Correct
I would expect the volumes ./html and ./db to be backed up as well. So all in all, I would expect the whole
/home/user/docker/worpdress folder to be backed up. As suggested, I may have it wrongly configured, fyi.
Gotcha, I'm guessing the nautical-backup.group would work, but the containers are getting skipped because of a name mismatch.
- The container
wordpress-dbhas a folder nameddbso Nautical is skipping it since it doesn't match the name. - The container
wordpress-bloghas a folder namedhtmlso Nautical is skipping it since it doesn't match the name.
You have two options that can fix it:
Option 1
Use overrides.
wordpress:
# Rest of config
volumes:
- ./html:/var/www/html
# /\ We are matching this
labels:
- "nautical-backup.group=wp"
- "nautical-backup.override-source-dir=html"
db:
# Rest of config
volumes:
- ./db:/var/lib/mysql
# /\ We are matching this
labels:
- "nautical-backup.group=wp"
- "nautical-backup.override-source-dir=db"
Option 2
Change the mounted directory in for your containers to match their names
Of course you will need to move the data over from the old folder to the new one.
# Example command (not exact)
cd <where your source dir is>
mv db wordpress-db
mv html wordpress-blog
wordpress:
# Rest of config
container_name: wordpress-blog
volumes:
- ./wordpress-blog:/var/www/html # <--- Change this
# /\ Now we match the "container_name"
labels:
- "nautical-backup.group=wp"
db:
# Rest of config
container_name: wordpress-db
volumes:
- ./wordpress-db:/var/lib/mysql # <--- Change this
# /\ Now we match the "container_name"
labels:
- "nautical-backup.group=wp"
Hopefully this help :)
Thank you for your feedback. Firstly I have tried Option 1 overrides so...
nautical backup docker-compose.yml
version: '3'
services:
nautical-backup:
image: minituff/nautical-backup:latest
container_name: nautical-backup
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/user/docker:/app/source
- /home/user/dockerbackup:/app/destination
environment: # Optional variables
- LOG_LEVEL=DEBUG
- CRON_SCHEDULE=21 9 * * *
- SKIP_CONTAINERS=nautical-backup
restart: unless-stopped
and
wordpress docker-compose.yml
version: '3.1'
services:
wordpress:
image: wordpress:6.5
container_name: wordpress-blog
restart: unless-stopped
ports:
- 80:80
environment:
WORDPRESS_DB_HOST: xxxx
WORDPRESS_DB_USER: xxxx
WORDPRESS_DB_PASSWORD: xxxx
WORDPRESS_DB_NAME: xxxx
volumes:
- ./html:/var/www/html
labels:
- "nautical-backup.group=wp"
- "nautical-backup.override-source-dir=html"
db:
image: mysql:8.0
container_name: wordpress-db
restart: always
environment:
MYSQL_DATABASE: xxxx
MYSQL_USER: mxxxx
MYSQL_PASSWORD: xxxx
MYSQL_RANDOM_ROOT_PASSWORD: xxxx
volumes:
- ./db:/var/lib/mysql
labels:
- "nautical-backup.group=wp"
- "nautical-backup.override-source-dir=db"
The group wp is detected:
INFO: Backing up group: wp
However, log output seems to indicate nautical backup is not detecting folders db or html and so not backing up. Longer log below:
DEBUG: REPORT_FILE: true
INFO: Nautical Backup Version: 2.3.3
DEBUG: Built for the platform: linux/arm64
DEBUG: Perparing enviornment variables...
DEBUG: Found defaults.env
DEBUG: CRON_SCHEDULE_ENABLED: true
DEBUG: BACKUP_ON_START: false
DEBUG: USE_DEST_DATE_FOLDER: false
DEBUG: DEST_DATE_PATH_FORMAT: date/container
DEBUG: DEST_DATE_FORMAT: %Y-%m-%d
DEBUG: USE_DEFAULT_RSYNC_ARGS: true
DEBUG: REQUIRE_LABEL: false
DEBUG: REPORT_FILE_LOG_LEVEL: INFO
DEBUG: REPORT_FILE_ON_BACKUP_ONLY: true
DEBUG: KEEP_SRC_DIR_NAME: true
DEBUG: EXIT_AFTER_INIT: false
DEBUG: LOG_RSYNC_COMMANDS: false
DEBUG: RUN_ONCE: false
DEBUG: SOURCE_LOCATION: /app/source
DEBUG: DEST_LOCATION: /app/destination
DEBUG: TEST_MODE: -1
DEBUG: HTTP_REST_API_ENABLED: true
DEBUG: HTTP_REST_API_USERNAME: admin
DEBUG: HTTP_REST_API_PASSWORD: xxxx
DEBUG: ADDITIONAL_FOLDERS_WHEN: before
DEBUG: NAUTICAL_DB_PATH: /config
DEBUG: NAUTICAL_DB_NAME: nautical-db.json
DEBUG: SELF_CONTAINER_ID: xxxxxxxxxxxx
DEBUG: Installing CRON schedule: 21 9 * * *
DEBUG: Verifying source directory '/app/source'...
DEBUG: Verifying destination directory '/app/destination'...
INFO: Initializing database at '/config/nautical-db.json'...
INFO: Creating Database at path: '/config/nautical-db.json'...
INFO: Database initialized at '/config/nautical-db.json'...
DEBUG: Installing nautical backup script...
INFO: API listening on port 8069...
INFO: Initialization complete. Awaiting CRON schedule: 21 9 * * *
DEBUG: Verifying source directory '/app/source'...
DEBUG: Verifying destination directory '/app/destination'...
INFO: Starting backup...
INFO: Processing 10 containers...
DEBUG: Containers: nautical-backup, wordpress-blog, wordpress-db [...]
DEBUG: Skipping nautical-backup based on name
INFO: Backing up group: wp
DEBUG: wordpress-db - Source directory '/app/source/db' does not exist. Skipping
DEBUG: wordpress-blog - Source directory '/app/source/html' does not exist. Skipping
DEBUG: Container wordpress-db was not stopped. No need to start.
DEBUG: Container wordpress was not stopped. No need to start.
So it wasn't backed up,source folder
/home/user/docker/wordpress
is not showing under
/home/user/dockerbackup
as
/home/user/dockerbackup/wordpress
I will try Option 2 later today and report. Thank you for your time and effort.
Note: folder db is owned by 999:root and folder html by www-data:www-data just in case
Second test with Option 2
Same nautical backup docke-compose.yml
wordpress docker-compose.yml
version: '3.1'
services:
wordpress:
image: wordpress:6.5
container_name: wordpress-blog
restart: unless-stopped
ports:
- 80:80
environment:
WORDPRESS_DB_HOST: xxxx
WORDPRESS_DB_USER: xxxx
WORDPRESS_DB_PASSWORD: xxxx
WORDPRESS_DB_NAME: xxxx
volumes:
- ./wordpress-blog:/var/www/html
labels:
- "nautical-backup.group=wp"
db:
image: mysql:8.0
container_name: wordpress-db
restart: always
environment:
MYSQL_DATABASE: xxxx
MYSQL_USER: mxxxx
MYSQL_PASSWORD: xxxx
MYSQL_RANDOM_ROOT_PASSWORD: xxxx
volumes:
- ./wordpress-db:/var/lib/mysql
labels:
- "nautical-backup.group=wp"
user@server:~/docker/wordpress $ ls -alh
-rw-r--r-- 1 user user 1.4K Jul 30 22:13 docker-compose.yml
drwxr-xr-x 6 www-data www-data 4.0K Jul 30 09:16 wordpress-blog
drwxr-xr-x 9 999 root 4.0K Jul 30 20:22 wordpress-db
This time all containers stopped before having nautical-backup run, so only nautical backup and the 2 wodpress containers were running. Log:
DEBUG: REPORT_FILE: true
INFO: Nautical Backup Version: 2.3.3
DEBUG: Built for the platform: linux/arm64
DEBUG: Perparing enviornment variables...
DEBUG: Found defaults.env
DEBUG: CRON_SCHEDULE_ENABLED: true
DEBUG: BACKUP_ON_START: false
DEBUG: USE_DEST_DATE_FOLDER: false
DEBUG: DEST_DATE_PATH_FORMAT: date/container
DEBUG: DEST_DATE_FORMAT: %Y-%m-%d
DEBUG: USE_DEFAULT_RSYNC_ARGS: true
DEBUG: REQUIRE_LABEL: false
DEBUG: REPORT_FILE_LOG_LEVEL: INFO
DEBUG: REPORT_FILE_ON_BACKUP_ONLY: true
DEBUG: KEEP_SRC_DIR_NAME: true
DEBUG: EXIT_AFTER_INIT: false
DEBUG: LOG_RSYNC_COMMANDS: false
DEBUG: RUN_ONCE: false
DEBUG: SOURCE_LOCATION: /app/source
DEBUG: DEST_LOCATION: /app/destination
DEBUG: TEST_MODE: -1
DEBUG: HTTP_REST_API_ENABLED: true
DEBUG: HTTP_REST_API_USERNAME: admin
DEBUG: HTTP_REST_API_PASSWORD: xxxx
DEBUG: ADDITIONAL_FOLDERS_WHEN: before
DEBUG: NAUTICAL_DB_PATH: /config
DEBUG: NAUTICAL_DB_NAME: nautical-db.json
DEBUG: SELF_CONTAINER_ID: xxxxxxx
DEBUG: Installing CRON schedule: 17 22 * * *
DEBUG: Verifying source directory '/app/source'...
DEBUG: Verifying destination directory '/app/destination'...
INFO: Initializing database at '/config/nautical-db.json'...
INFO: Creating Database at path: '/config/nautical-db.json'...
INFO: Database initialized at '/config/nautical-db.json'...
DEBUG: Installing nautical backup script...
INFO: API listening on port 8069...
INFO: Initialization complete. Awaiting CRON schedule: 17 22 * * *
DEBUG: Verifying source directory '/app/source'...
DEBUG: Verifying destination directory '/app/destination'...
INFO: Starting backup...
INFO: Processing 3 containers...
DEBUG: Containers: nautical-backup, wordpress-blog, wordpress-db
DEBUG: Skipping nautical-backup based on name
INFO: Backing up group: wp
DEBUG: wordpress-db - Source directory '/app/source/wordpress-db' does not exist. Skipping
DEBUG: wordpress-blog - Source directory '/app/source/wordpress-blog' does not exist. Skipping
DEBUG: Container wordpress-db was not stopped. No need to start.
DEBUG: Container wordpress-blog was not stopped. No need to start.
DEBUG: Containers completed:
DEBUG: Containers skipped: wordpress-db, wordpress-blog, nautical-backup
INFO: Success. 0 containers backed up! 3 skipped.
So wordpress-db and wordpress-blog were not backed up. I may run more tests, as needed.
Hey @mixpc, thanks for testing this out.
For both options 1 and 2, did you mount the directories inside the Nautical contianer too?
# docker-compose.yml for nautical backup
version: '3'
services:
nautical-backup:
image: minituff/nautical-backup:latest
container_name: nautical-backup
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/user/docker:/app/source
- /home/user/dockerbackup:/app/destination
- ./wordpress-blog:/app/source/wordpress-blog # Option 2
- ./wordpress-db:/app/source/wordpress-db # Option 2
environment: # Optional variables
- LOG_LEVEL=DEBUG
- CRON_SCHEDULE=05 21 * * *
- SKIP_CONTAINERS=nautical-backup
restart: unless-stopped
What is the absolute path of ./wordpress-blog and ./wordpress-db? I can give you a better recommendation with that information.
Ah, no. Here are the full paths:
/home/user/docker/wordpress/wordpress-blog
/home/user/docker/wordpress/wordpress-db
All docker containers are located within
/home/user/docker
Ohhh, I think I get it, its because you have wordpress-blog and wordpress-db nested inside the wordpress folder.
Can you move them up to the root of the /home/user/docker folder like this:
/home/user/docker/wordpress-blog
/home/user/docker/wordpress-db
OR
You could update the mounts in the Nautical config like this:
# docker-compose.yml for nautical backup
version: '3'
services:
nautical-backup:
image: minituff/nautical-backup:latest
container_name: nautical-backup
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/user/docker:/app/source
- /home/user/dockerbackup:/app/destination
- /home/user/docker/wordpress/wordpress-blog:/app/source/wordpress-blog # HERE
- /home/user/docker/wordpress/wordpress-db:/app/source/wordpress-db # AND HERE
environment: # Optional variables
- LOG_LEVEL=DEBUG
- CRON_SCHEDULE=05 21 * * *
- SKIP_CONTAINERS=nautical-backup
restart: unless-stopped
@Minituff Success! My nesting was causing the issue... Alas, I took for granted nesting would work as it allowed me have a clearer folder structure. So, thank you for the piece of advice! BTW, do you think nesting could work? Maybe it's too much trouble for little advantage.
Hey @vg8020 Glad it's working!!
And I just pushed out an update for you that will enable this feature.
What you'll need to do is use the Overrides and append the nested path, like this.
# wordpress docker-compose.yml
version: '3.1'
services:
wordpress:
image: wordpress:6.5
container_name: wordpress-blog
restart: unless-stopped
volumes:
- ./wordpress-blog:/var/www/html
labels:
- "nautical-backup.group=wp"
- "nautical-backup.override-source-dir=wordpress/wordpress-blog" # <----- Update this
db:
image: mysql:8.0
container_name: wordpress-db
restart: always
volumes:
- ./wordpress-db:/var/lib/mysql
labels:
- "nautical-backup.group=wp"
- "nautical-backup.override-source-dir=wordpress/wordpress-db" # <----- Update this