[Feature Request] Scheduled Backups

[CameronJGrant]

Have you read the Important information text above

• [x] Yes i did

Current behavior

Server saves aren't backed up.

Desired behavior

The save files for the server should be backed up by some cron schedule, if desired, and controlled through environmental variables. Especially with the issues being reported about save files being lost.

Check out this Valheim server for a good example: https://github.com/lloesche/valheim-server-docker

BACKUPS=true BACKUPS_CRON=0 * * * * BACKUP_DIRECTORY=/something etc...

Basically, we just set up a cronjob in the container to run the admin /Save command to do a forced save, then copy that save to the backup directory and date it.

Save folder: /palworld/Pal/Saved/SaveGames/0/9D3D178236CA4DCD977E978D94FFD633/Players

Links to screenshots

No response

To Reproduce

NA

Software setup

NA

Hardware setup

NA

Additional context

No response

[jammsen]

Hey @CameronJGrant - I like the idea, would look into this tomorrow. Few question though:

• Dont you mean a "tar.gz" of Saved?
• How would you run the /Save command inside the container? How would you inject that in the process? The RCON tool is outside of the container
• Are we now mixing internal and external work?
• Could you give me more details on your thoughts so i get a better understanding of the workflow you imagine?
• I will look into https://github.com/lloesche/valheim-server-docker tomorrow, its no 01:19 in the morning 🥱

[uncaught]

You'd need a whole process manager to also have cron run in the container. Might be a bit too much for now because your entrypoint eventually delegates to the palserver directly. Valheim used something like that, but they had years of development ;)

I'd just add an example to the readme how to create backups with cron yourself.

Like backup.sh:

#!/bin/bash
# dir should be absolute for cron or you might end up anywhere:
cd /your/palworld/directory
mkdir -p backups
tar -czf backups/${something-with-date}.tar.gz game/Pal/Saved

And just call that via crontab.

[jammsen]

You'd need a whole process manager to also have cron run in the container. Might be a bit too much for now because your entrypoint eventually delegates to the palserver directly. Valheim used something like that, but they had years of development ;)

I'd just add an example to the readme how to create backups with cron yourself.

Like backup.sh:

#!/bin/bash
# dir should be absolute for cron or you might end up anywhere:
cd /your/palworld/directory
mkdir -p backups
tar -czf backups/${something-with-date}.tar.gz game/Pal/Saved

And just call that via crontab.

Im already done implementing a solution, finishing the last tests and then i will push the changes.

[jammsen]

Valheim used something like that, but they had years of development ;)

I know and i would rather stay away from unnecessary bloating the image with an additional process manager like supervisord, im very happy i got it out again in my Sons Of The Forest and The Forest Dedicated.

Im using now a solution thats more container oriented and dont need all that stuff like cron does.

[jammsen]

Hey @uncaught @CameronJGrant - I have implemented a backup-solution that does hourly backups in your game directory. If you want to change settings, you can enable or disable this and you can change the cron-expression, meaning the interval it happens. Therest is hardcoded on purpose and i prefer it that way. Too much open config can result in recursive backups and an early full harddisk and people beeing mad at me. I dont want that.

Please go ahead and try it out .

[abdonkov]

The backups look great, but could you also implement a configurable backup retention. Like for example retaining the last 10 backups, or last day (timeframe) of backups, whichever is easier. So I wouldn't need to implement my own cron just to handle the backup management, and maybe less problems with directory access. Also you will definitely wont have complains for a full hard disk with a sensible default...

[CameronJGrant]

Excellent work @jammsen!

Hey @CameronJGrant - I like the idea, would look into this tomorrow. Few question though:

• Dont you mean a "tar.gz" of Saved?

palworld/Pal/Saved would be better. I only had a few minutes to dig around the container yesterday and I wasn't sure if palworld/Pal/Saved/Config had anything that needed saving. I agree with your solution.

• How would you run the /Save command inside the container? How would you inject that in the process? The RCON tool is outside of the container

You can run the RCON commands using something like netcat directly from a shell script within the container. We can include anything you like in your "backupmanager.sh".

• Are we now mixing internal and external work?

Everything should be contained in one image. It's all internal.

• Could you give me more details on your thoughts so i get a better understanding of the workflow you imagine?

Looks like you nailed it. The only things I would add in future, (and maybe I will create a PR for this since you already have your hands full) is to still call /Save explicitly using the RCON commands before backing up. Some games broadcast a warning before saving because a save can interrupt gameplay, but I don't think Palworld has that issue.

• I will look into https://github.com/lloesche/valheim-server-docker tomorrow, its no 01:19 in the morning 🥱

I think we are on very different time zones.

You'd need a whole process manager to also have cron run in the container. Might be a bit too much for now because your entrypoint eventually delegates to the palserver directly. Valheim used something like that, but they had years of development ;)

You don't need all that. You just need cron running in the container. I would have just installed cron via apt-get but @jammsen came up with a far more elegant solution with supercronic.

[CameronJGrant]

The backups look great, but could you also implement a configurable backup retention. Like for example retaining the last 10 backups, or last day (timeframe) of backups, whichever is easier. So I wouldn't need to implement my own cron just to handle the backup management, and maybe less problems with directory access. Also you will definitely wont have complains for a full hard disk with a sensible default...

Absolutely right as well.

Keep within retention period(in days):

find /palworld/backups -name 'saved-*.tar.gz' -type f -mtime +$RETENTION_DAYS -exec rm {} \;

Or it might be better to just keep the latest n files in case you shut down the server for an extended period of time and set it back up later, all backups would get wiped. But people should be moving these backups offsite (to S3 or something) routinely anyway, so it's not a big deal.

Keep last N files:

cd /palworld/backups
ls -1t saved-*.tar.gz | tail -n +$(($KEEP_LAST_N_SAVES + 1)) | xargs -d '\n' rm -f --

[jammsen]

The backups look great, but could you also implement a configurable backup retention. Like for example retaining the last 10 backups, or last day (timeframe) of backups, whichever is easier. So I wouldn't need to implement my own cron just to handle the backup management, and maybe less problems with directory access. Also you will definitely wont have complains for a full hard disk with a sensible default...

@abdonkov - Im open for suggestions/ideas/code examples, everything im coming up has up-sides and drawbacks. I thought at least deliver the function to test first 😭 - Keeping only days, will kill inactive server with valid saves on a bug, therefore no backups anymore. Keeping the last 1?3?10? How to decide? How to do?

Looks like you nailed it. The only things I would add in future, (and maybe I will create a PR for this since you already have your hands full) is to still call /Save explicitly using the RCON commands before backing up. Some games broadcast a warning before saving because a save can interrupt gameplay, but I don't think Palworld has that issue.

@CameronJGrant - I think i have even seen something like this coded, dont remember right now where and what the context was 😭 but i agree send /save and /broadcast "Server-Save in progress ..." would be cool. -> https://tech.palworldgame.com/server-commands

I think we are on very different time zones.

Germany!

You don't need all that. You just need cron running in the container. I would have just installed cron via apt-get but @jammsen came up with a far more elegant solution with supercronic.

😊 ❤️

Im open to ideas on the "Rentention-Policy" - GO NUTS and help me out please 😄

[CameronJGrant]

@abdonkov - Im open for suggestions/ideas/code examples, everything im coming up has up-sides and drawbacks. I thought at least deliver the function to test first 😭 - Keeping only days, will kill inactive server with valid saves on a bug, therefore no backups anymore. Keeping the last 1?3?10? How to decide? How to do?

Once we figure out how to use RCON in script's more easily we should use the /ShowPlayers command to check if anyone's in game before saving so that we aren't backing up unless someone is actually in there doing something. Then add a grace period after the server empties that backups can be created in.

I suggest we do that and allow users to choose their retention policy.

Possible environment variables:

Variable	Description	Default Value	Allowed Value
BACKUP_ENABLED	Backup function, creates backups in your game directory	true	false/true
BACKUP_CRON_EXPRESSION	Needs a Cron-Expression - See https://github.com/aptible/supercronic#crontab-format or https://crontab-generator.org/	0 * * * * (meaning every hour)	Cron-Expression
BACKUP_MAX_COUNT	The maximum number of backup files to keep	5	Integer value
BACKUP_MAX_AGE	The maximum age for a backup file before it is deleted (in days)	30	Integer value (days)
BACKUP_IF_IDLE	Perform backup only if the server is idle	false	false/true
BACKUP_GRACE_PERIOD	Time to wait before starting the backup after the scheduled time (in minutes)	10	Integer value (minutes)
BACKUP_ZIP	Whether to compress the backup files into ZIP format	true	false/true

Weird side note. It saves backups of local data to the client's PC. I'm not sure if you could fix a server instance with that though. https://docs.google.com/document/u/0/d/e/2PACX-1vR-lb9R2fh-IVK870PL82U071i-9-C_Y7YyroUN8EoD_34rbpKQzjsCejCHRb4XMGncY9HkI0OxQv1F/pub?pli=1#h.xl4nmt5nrqb9

@CameronJGrant - I think i have even seen something like this coded, dont remember right now where and what the context was 😭 but i agree send /save and /broadcast "Server-Save in progress ..." would be cool. -> https://tech.palworldgame.com/server-commands

Maybe it was here? https://gist.github.com/mbasaglia/62d7fef83304f4b3ca90

Germany!

Cool!

Might I also suggest opening a develop branch?

[abdonkov]

Im open to ideas on the "Rentention-Policy" - GO NUTS and help me out please 😄

I mean if you don't want to decide how much backups to retain you can always set it to 0, meaning the retention by count won't be active. This will also require a retention days parameter as a fallback, like 2, 3 days. Something small so with the default options, the backups won't use too much space and you wont get complains 😄

If you are looking for inspiration, you could always see in the Valheim server docker container what parameters are implemented, which are mostly the ones that @CameronJGrant suggested.

[jammsen]

Once we figure out how to use RCON in script's more easily we should use the /ShowPlayers command to check if anyone's in game before saving so that we aren't backing up unless someone is actually in there doing something. Then add a grace period after the server empties that backups can be created in.

I suggest we do that and allow users to choose their retention policy.

#Overkill 🤣

Maybe it was here? https://gist.github.com/mbasaglia/62d7fef83304f4b3ca90

No i could be here https://github.com/jammsen/docker-conanexiles/blob/master/src/usr/bin/conanexiles_controller#L103 it wasnt from the perspective of my build a thing, it was more of the experience of using person, like "im the admin of a dedicated server and host this for my friends and i find the feature ive not seen like this before very cool". Sadly i didnt fork it, because i didnt knew i would need it for something.

Might I also suggest opening a develop branch?

Yes, hopefully tomorrow.

[jammsen]

If you are looking for inspiration, you could always see in the Valheim server docker container what parameters are implemented, which are mostly the ones that @CameronJGrant suggested.

Just to be on the safe side, you mean this one right? https://github.com/lloesche/valheim-server-docker

[doodoori2]

Hi guys, where is the "Rentention-Policy" you mentioned? is there any discord or ideas page? how can I read the thread?

[jammsen]

@doodoori2 Read this https://github.com/jammsen/docker-palworld-dedicated-server/issues/27#issuecomment-1906980779 and realize you are already here ;)

[jammsen]

Im open to ideas on the "Rentention-Policy" - GO NUTS and help me out please 😄

I mean if you don't want to decide how much backups to retain you can always set it to 0, meaning the retention by count won't be active. This will also require a retention days parameter as a fallback, like 2, 3 days. Something small so with the default options, the backups won't use too much space and you wont get complains 😄

If you are looking for inspiration, you could always see in the Valheim server docker container what parameters are implemented, which are mostly the ones that @CameronJGrant suggested.

I think i will just go with a keep N files strategy, no need to overcomplicate everything.

Keep last N files:

cd /palworld/backups
ls -1t saved-*.tar.gz | tail -n +$(($KEEP_LAST_N_SAVES + 1)) | xargs -d '\n' rm -f --

Actually that looks already nice to get a better understanding and get started

[abdonkov]

Just to be on the safe side, you mean this one right? https://github.com/lloesche/valheim-server-docker

Yes, that one.

[jammsen]

Hey @CameronJGrant - Sorry for the delay i finaly got around to finish this. But Bugs > Features, sorry.

Ive added now a switch to turn it on or off and a number to keep with even you shell-code that i really liked. Works like a charm. Thanks for your contribution!

[jammsen]

Im closing this issue as resolved, feel free to try it out and reopen again if needed.

If you like this project, please consider giving this repo and the docker-hub-repo a Star.

[CameronJGrant]

For sure I'll give you a star! Thanks for the hard work! Ironically I just had my server wipe lol. Luckily I had a few manual backups so it wasn't a big deal.

[CameronJGrant]

Hey @jammsen. There isn't a reopen issue button so maybe it's turned off?

Anyway, small thing. I found out the hard way that if disk space fills in your server Palworld will delete your server file. It looks like, when it saves your game, it deletes your old file first, then saves the new one.

That said, I notice the defaults we have set are:

So I notice your defaults are:

Variable	Default
BACKUP_ENABLED	true
BACKUP_CRON_EXPRESSION	0 * * * *
BACKUP_RETENTION_POLICY	false
BACKUP_RETENTION_AMOUNT_TO_KEEP	30

This means, by default, we are backing up every hour and not removing any old backups. In a year people will have almost 9000 backups. It only comes to about 18GB but a lot of people run small servers. We might be sitting on a ticking time bomb as is.

I suggest having the retention policy on by default and set the time to once per day. Then 30 days are covered. (Since 30 at hourly is only one day).

[jammsen]

For sure I'll give you a star! Thanks for the hard work! Ironically I just had my server wipe lol. Luckily I had a few manual backups so it wasn't a big deal.

That sounds horrible and good at the same time 🤣

Hey @jammsen. There isn't a reopen issue button so maybe it's turned off?

OH DANG SORRY - https://stackoverflow.com/a/21333938

I suggest having the retention policy on by default and set the time to once per day. Then 30 days are covered. (Since 30 at hourly is only one day).

I get your point and i would have done that when i introduced BOTH features at the same time, it wasnt the case for this, it was incremental. Thats why i had to really hard think about it and didnt wanted people to have to loose savegames they didnt backup before, thats why its on false by default. But i get your point, thats why i rewrote the Setup-6-Steps another time after splitting files to make sure people CONFIG it to their liking you know?

[jammsen]

Thanks for the hard work!

No! Thank YOU for your contribution Keanu-Wave! ❤️