docker-minecraft-bedrock-server
docker-minecraft-bedrock-server copied to clipboard
itzg
feature RQ -- Bedrock Worlds backup
My server's been been working great on my MacMini, thanks to this repo! With few friends we've started building a village, with people connecting at various times. It would be a shame if all wonderful builds they create got destroyed.
Bedrock server has the built-in backup functionality which works as I tested. But it is far from automated.
It involves tying in the server console save hold, save query, copy world files and save resume.
I was wondering if it would be somehow possible to automate it. Run with an interval. Perhaps a shell script in the container sending mentioned commands to the Bedrock console and then copying files to an external volume.
I could work on this but don’t have clue where to start...
Maybe I could use something like Expect?
I agree that it would be really nice to automate that process, but yes it would need to be something that could inspect the save query and react accordingly.
Right now the server process is wrapped by this helper tool I developed https://github.com/itzg/entrypoint-demoter
That tool is currently generic to any type of containerized process; however, I'm thinking I need to spin off a bedrock server specific version of that. With that it could expose bedrock-specific operations such as doing a backup.
Enhancing that Go-based tool might have been more than what you were offering, but that's where I would see the enhancement happening. I have been wanting to expose console access via a web API anyway, so I might prioritize working on both of these changes soon myself.
btw.. If you play MC yourself it would be a pleasure to invite u to our lil server, it’s great fun. Let me know!
I'm kind of a solitary LAN-server player, but yeah I should really get out there in the "world" ;) Shoot me your server address in email ([email protected]) or DM me on Discord and I'll jump on sometime.
I’ve created a little script that seems to be working fine on 1.16.1, thought I share.
It requires expect and can be run with cron or lunchd every half hour or so.
(You might need to verify your docker path by running which docker)
#!/usr/bin/expect --
set world_dir <dirs>/worlds
set docker_container <container name>
set world_name <world name>
set backup_dest “<dirs>"
set timestamp [timestamp -format {%Y-%m-%d--%H-%M}]
set world_compress_timeout 240
set world_arch_mv_timeout 30
# log_file <path>backup-world.log # the log file might get large quickly!
spawn /usr/local/bin/docker attach --detach-keys=Q $docker_container
expect "DEBU*”
while true {
send "save hold\r"
expect "Saving...*" { break }
expect "The command is already running*" { break }
}
sleep 1
while true {
send "save query\r"
expect "Data saved.*" { break }
}
while true {
send "Q"
expect eof { break }
}
spawn bash
send "cd \"$world_dir\"\r"
expect "*$ "
set timeout $world_compress_timeout
send "tar -zcf $world_name.$timestamp.tgz $world_name\r"
expect "*$ "
set timeout $world_arch_mv_timeout
send "mv $world_name.$timestamp.tgz \"$backup_dest\"\r"
expect "*$ "
send "exit\r"
expect eof
set timeout 10
spawn /usr/local/bin/docker attach --detach-keys=Q $docker_container
expect "DEBU*"
while true {
send "save resume\r"
expect "Changes to the level are resumed.*" { break }
}
while true {
send "Q"
expect eof { break }
}
exit
Here is my simple backup-script. Works since one year!
#!/bin/bash docker-compose -f "/home/daniel/dropbox/data/docker/3-minecraft/docker-compose.yml" down
_day="$(date +'%A')" _file="minecraft-data-${_day}.tar" _path="/home/daniel/dropbox/data/docker-backup/${_file}"
tar -P -cf ${_path} --exclude='bedrock_server*' /home/daniel/docker-data/minecraft --overwrite
docker-compose -f "/home/daniel/dropbox/data/docker/3-minecraft/docker-compose.yml" up -d --build
I'm sort of new to Minecraft, and I'm investigating setting up a Bedrock server to run on our LAN for our kids and a few of their friends. I don't have a clear understanding of the process of backing up worlds. When you speak of commands like "save hold" and "save query", is that to facilitate backing up a running server? Would it be an acceptable workaround to periodically stop the server, make a backup of the worlds folder and then start the server, assuming this is done when the server is definitely not in use?
@wedgef5 yes, doing the backup while the server is stopped would be a great alternative.
Just wanted to comment that I've been playing around with a tool that can do these sort of backups. I started by creating a CLI tool I could run via systemd on the docker host, and I recently containerized it with a bit of help from reading through the docker-mc-backup scripts. It is able to backup a running server just fine, but it does require access to the host's docker.sock file in order to accomplish this.
It's a bit rough around the edges at the moment because it's not what I'd consider fully containerized and so the config file contains values it doesn't actually need, but it is what I've been using for about 3 weeks now (via systemd) and the last few days (via docker) on my own two-server host.
Docker Hub: https://hub.docker.com/r/kaiede/minecraft-bedrock-backup GitHub: https://github.com/Kaiede/docker-minecraft-bedrock-backup CLI Tool: https://github.com/Kaiede/BedrockifierCLI
Awesome. @Kaiede when you're ready for it, I would be happy to add a "Backups solutions" section to the README that links to your image.
I've done the cleanup I want to do around the configuration. I think I've managed to get it to the "minimum viable product" state that I'm happy with, and the changes have shown to be stable on my home server. So feel free to add the reference.
Mostly worried that the documentation needs some revision, but at this point, will have to see where folks get hung up on configuring things.
There it is:
https://github.com/itzg/docker-minecraft-bedrock-server#solutions-for-backing-up-data
This is awesome. I’m just wondering how that compares performance-wise vs just making dumps of a world via MC server’s CLI and just tar-ing them? My server runs on a quite low-RAM machine stretched up to limit so wondering if another docker container would take-up much resources.
Containers don't add any memory overhead, so @Kaiede 's backup container would only occupy the memory for the bash script to sleep most of the time. Since the tool is compiled from Swift, I'm guessing it has very low memory footprint for those moments it runs.
The processes running in the container on my server are eating about 4KB of RAM total while sleeping. When backing up my two containers and trimming the backup list (one 80MB world, and a 300MB world), the peak RAM stayed under 60MB (RSS) and took less than one second (NVMe SSD). I don’t see this growing a lot with larger worlds, since the buffers are fixed in size.
Since the tool is compiled from Swift, I'm guessing it has very low memory footprint for those moments it runs.
Depends on what you compare it to. Something like the POSIX suite of tools are hard to beat. But compared to say, the node process for manymine, or dockerd, it uses less. The Swift runtime isn’t quite as good as the C/C++ runtime in memory usage.
I've done the cleanup I want to do around the configuration. I think I've managed to get it to the "minimum viable product" state that I'm happy with, and the changes have shown to be stable on my home server. So feel free to add the reference.
Mostly worried that the documentation needs some revision, but at this point, will have to see where folks get hung up on configuring things.
Just setting a new world for my server so decided to finally give it a go :-) Hunged-up on paths config... :-|
docker-compose for my server for reference:
version: '3.4'
volumes:
emer2_vol:
external: true
services:
bds2:
container_name: emer2
image: itzg/minecraft-bedrock-server
environment:
# blah...
ports:
- 19133:19132/udp
- 39134:39134
- 39134:39134/udp
- 39135:39135
- 39135:39135/udp
sysctls:
net.ipv4.ip_local_port_range: 39134 39135
volumes:
- emer2_vol:/data
stdin_open: true
tty: true
restart: unless-stopped
backup:
image: kaiede/minecraft-bedrock-backup
name: emer2_backup
restart: always
depends_on:
- "bds2"
environment:
BACKUP_INTERVAL: "3h"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /opt/bedrock/backups:/backups
- /opt/bedrock/server:/server
I would like to save the backup in e. g. /Volumes/Storage/mine/emer2/backup/world. Do I need to create an external volume and define it in docker-compose, similarly as I do for the data volume? or do I just replace the /opt/bedrock/backups bit. Also is the external data volume going to work with the above setup?
(also not sure if should just post this in https://github.com/Kaiede/docker-minecraft-bedrock-backup)
@MattrCoUk since you designated emer2_vol external, then yes you'll need to create that yourself. You also need to attached emer2_vol to the backup container at /data in order to point it at the content to backup.
Thanks for working on this @Kaiede
Will this solution also work on a Win10 host with appropriate path changes?
I would like to save the backup in e. g. /Volumes/Storage/mine/emer2/backup/world. Do I need to create an external volume and define it in docker-compose, similarly as I do for the data volume? or do I just replace the /opt/bedrock/backups bit. Also is the external data volume going to work with the above setup?
@MattrCoUk : My example for the backup is mostly there to give an idea what's possible. Whatever you map to /backups is where backup files get written to, and where config.json is expected to be, by default. So replace /opt/bedrock/backups with whatever you want so you can access things externally, such as /Volumes/Storage/mine/emer2/backup/world. I will say one thing though, is that I have never quite figured out how to deal with named volumes for restoring backups. It's honestly easier to restore while the server and backup containers are shut down, which makes it harder to access named volumes (something I never figured out how to do). So I'd generally recommend using external volumes for both containers so it's possible to shut down the containers, unzip the *.mcworld file into the world folder, and then start the containers back up.
Will this solution also work on a Win10 host with appropriate path changes?
@wedgef5 : In principle, it should, the same way running the container on macOS works. They both have to virtualize Linux to run either the backup container or the bedrock server container. So just keep in mind that paths inside the container are still going to be *nix paths.
So I’ve set it up on both of my worlds and it seems to be working splendidly!
here’s my docker-compose for reference uses 2 external volumes
version: '3.4'
volumes:
# create volumes before running docker-compose:
# docker volume create --opt device=path/to/server/data --opt o=bind bs_data_volume --opt type=none
# docker volume create --opt device=path/to/backup --opt o=bind bs_backup_volume --opt type=none
bs_data_volume:
external: true
bs_backup_volume:
external: true
services:
bedrock_server:
container_name: bedrock_server
image: itzg/minecraft-bedrock-server
volumes:
- bs_data_volume:/data
# ... etc.
bs_backup:
container_name: bs_backup
image: kaiede/minecraft-bedrock-backup
restart: always
depends_on:
- "bedrock_server"
environment:
BACKUP_INTERVAL: "3h"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- bs_backup_volume:/backups
- bs_data_volume:/server
notes:
server's container_name and the service name has to be the same for backups to work correctly would be nice to have an option to run backups on schedule (such as 6:00AM, 12:30PM etc.)
server's container_name and the service name has to be the same for backups to work correctly
docker-compose will name a container with a prefix if you don't specify the container_name manually. So if I have my compose file at: /opt/mygreatserver/docker-compose.yml and name the service bedrock_server the container winds up being named mygreatserver_bedrock_server, and other containers get the same prefix, effectively grouping them together. As long as the name docker itself sees is in the config.json, it will work.
So yeah, naming it explicitly is a good way to keep things easier to remember and work with for sure.
would be nice to have an option to run backups on schedule (such as 6:00AM, 12:30PM etc.)
Feel free to open an issue on my github repo for this. I intentionally left it out of the first iteration because:
- Making users enable cron on the docker host is effectively a support time sink.
- Integrating something cron-like in the container itself is pretty heavyweight.
- I have ideas that would require turning the backup tool itself into an always-running service monitoring the bedrock server (triggering backups on logout/login). If I go that route, then cron behaviors really should be handled by this service directly.
I figured being able to trim the backups, and run them a bit more frequently would help address some of the need for cron scheduling in the short term at least.
I agree with @Kaiede about the hesitations for directly supporting cron schedules. At the risk of being a heavy answer to an easy question, but kubernetes CronJob workloads are a robust way to support scheduled container activities
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
I’ve noticed that on one of my servers the backup produces some additional files. Other server with a similar backup settings doesn’t have them. Wonder if those are important.
I’ve noticed that on one of my servers the backup produces some additional files.
Right now, when a backup is fired, all worlds detected in the worlds folder for a server are backed up. A “valid” world is one where there is a folder with “levelname.txt” inside, and the tool can extract a useful string from that file.
The main issue here is that since the tool is unaware of what world the server is using, or if the user might have swapped out the worlds between backups firing, it just backs it all up. This is something I can probably document better.
So what’s likely happening here is that you’ve got a second folder in your server’s worlds folder, and it’s levelname.txt file contains the string “PMC87f…”. I can’t really say anything about what created that folder in your case, but if you were to dig into your server directory and find that folder, and clean it up, these extra backups would stop being made. Being only 6KB in size, it almost seems like a world that was “created” but never properly named or logged into.
I’ve noticed that on one of my servers the backup produces some additional files.
Right now, when a backup is fired, all worlds detected in the worlds folder for a server are backed up. A “valid” world is one where there is a folder with “levelname.txt” inside, and the tool can extract a useful string from that file.
The main issue here is that since the tool is unaware of what world the server is using, or if the user might have swapped out the worlds between backups firing, it just backs it all up. This is something I can probably document better.
So what’s likely happening here is that you’ve got a second folder in your server’s worlds folder, and it’s levelname.txt file contains the string “PMC87f…”. I can’t really say anything about what created that folder in your case, but if you were to dig into your server directory and find that folder, and clean it up, these extra backups would stop being made. Being only 6KB in size, it almost seems like a world that was “created” but never properly named or logged into.
Ahhhh, I forgot that I had to change the server's name temporarily in order to verify it on planet.minecraft. It generated a new world..... Silly me! Great feature with backing up all worlds automatically btw.
Would anybody be able to help me understand the backup process a bit better? I've been reading about "save hold", "save query", screen, expect, etc. for a couple of hours now and still not quite comprehending everything.
Taking the docker commands (docker stop, docker start, docker attach) out of the picture; if we wanted to take a backup from INSIDE the running container is there a straightforward process for doing so?
@Kaiede I've looked at your BedrockifierCLI tool and that looks like it is used outside of the docker container, am I interpreting that correctly?
Thank you for any guidance or information.
Taking the docker commands (docker stop, docker start, docker attach) out of the picture; if we wanted to take a backup from INSIDE the running container is there a straightforward process for doing so?
Not really. You need to have access to the input/output of the server's console to issue commands to do this safely. So while you could write a script that runs inside the container, it would look a lot like a script that runs outside the container. The crux of taking a backup while the server runs relies on issuing the "save hold" and "save resume" commands. Those commands are there to tell the server to stop writing to disk for a moment so the backup can be made ("save hold"). Once the backup is made, you can issue "save resume" again to tell the server it's safe to write to disk again. Needing access to the server's console is what makes it a bit of a pain to get backup scripts working right.
The alternative is to shut down the server, make the backup and start it again instead, which achieves the same result, but brings down the server for anyone logged in while it does it. (This is the docker stop/start approach)
Docker itself isn't necessarily why it is complicated, and makes both approaches above a bit easier, to be honest.
I've looked at your BedrockifierCLI tool and that looks like it is used outside of the docker container, am I interpreting that correctly?
It currently assumes that the server to be backed up is in a docker container, but the tool itself can be used outside a container. I used it what way to backup my server container for a bit while I was working on the containerized version of the backup tool. But it doesn't support being used either in the same container as the server, or if the server isn't in a docker container at all.
The catch is that BedrockifierCLI can only really be built for Mac/Linux (Swift on Windows is still pretty experimental). So the containerized backup tool is honestly more compatible, and more "set and forget".
I've gotten this solution working on Win10 Pro. Thanks to @Kaiede for putting it together! Below is my docker-compose.yml which is using a mount to the host NTFS filesystem. Docker will issue a warning about this due to relatively poor performance of using Windows host FS mounts like this. I figure with the relatively low amount of IO that it shouldn't be an issue. I've also been investigating using a WSL2 Ubuntu instance to host the backup data, but I haven't actually tried it yet. It seems that it should be possible, though. My primary goal is to have the backed up worlds out of the container where they can be picked up by my cloud backup solution (iDrive). The easiest way to achieve that is for the backup to be on the host FS.
version: '3.4'
services:
bedrock:
image: itzg/minecraft-bedrock-server
container_name: bedrock_server
restart: always
environment:
EULA: "TRUE"
SERVER_NAME: "Home Server"
WHITE_LIST: "TRUE"
WHITE_LIST_USERS: "wedgef5"
GAMEMODE: creative
DIFFICULTY: normal
ONLINE_MODE: true
ports:
- 19132:19132/udp
volumes:
- bds:/data
stdin_open: true
tty: true
backup:
image: kaiede/minecraft-bedrock-backup
container_name: bedrock_backup
restart: always
depends_on:
- "bedrock"
environment:
BACKUP_INTERVAL: "3h"
TZ: "America/Chicago"
volumes:
- //var/run/docker.sock:/var/run/docker.sock
- G:/Minecraft-Backup:/backups
- bds:/server
volumes:
bds: {}
@wedgef5 Plumbing data around is always a pain, for sure.
Since I use B2 for my NAS backups already, I added a duplicati container to handle uploading my backups directory to a B2 container that houses my VM backup data. It looks like duplicati can be configured to upload to iDrive as well. That would let you wrap the whole thing up in either a Ubuntu VM or WSL2 setup.
I generally found a Hyper-V or VMWare VM was a bit more efficient (power, performance) than using Docker Desktop.
I got it working with a WSL2 Ubuntu instance. In the docker-compose.yml just substitute the following for the backups volume...
- \\wsl$$\Ubuntu\data\Minecraft-backup:/backups
I looked at Duplicati, and they don't appear to support "regular" iDrive backup. I think iDrive may have some other product that leverages S3 under the hood, but I found a Duplicati feature request ticket for iDrive, and they were unable to do it. iDrive does, fortunately, have a set of Perl scripts that can yield backup capability in a *nix VM.