nicotsx
Should we support traditional Databases?
Last week @altendorfme proposed a Pull Request to connect directly some traditional databases. Automatically dump databases and backup this instead of a folder.
While initially excited by the feature, I have later expressed a few concerns:
The more I tested the implementation the more I started thinking that maybe we are not the right tool to integrate Database backups.
- Databases are not real “volumes” and we are adding weird code branches to make up for this. We have to run dump executions or “hooks” before backups and after for cleaning up
- Since we create a new dump and back it up each time this goes a bit against the restic philosophy of incremental deduped backups
- The frontend functionality like volume size, file explorer, “mounted” status, file inclusions/exclusions make little sense and we are also writing code branches to hide these parts to the user.
- We increase the image size significantly with multiple clients that the user might not need
- Overall I think this complexifies the code too much while the user could simply run dump executions from a script on the host and backup that folder with Zerobyte.
I'm curious what the community thinks about it?
The concerns are valid. I think that even in the code you could open a separate databases area from the volumes, and clearly state the issues regarding dumps and single backups.
It would be possible to create a separate image but within the same workflow for those who want the database functionality, so as not to have an inflated main image.
Agreed, making a separate image can allow the core restic philosophy to stay the same, while allowing the more enterprise users to opt-in to database functionality.
I actively try to avoid hosting programs which require seperate databases because they can be cumbersome to backup and maintain. I switched to ironmount because it was user experience focussed and easier to use compared to other solutions as stated in the original reddit post, and if it did support database backup, it would remove some pain points of hosting database apps for many selfhosters. While recently few web-ui based database backup programs have emerged none of them are quite as friendly as zerobyte, and if I'm already using one program to backup, I'd prefer it to backup everything and not require a separate backup solution for some of the apps.
As for the size concerns, I doubt anyone would be bothered if the container image was few mb larger.
I support @altendorfme's suggestion: Just like volumes, databases should be a separate section like volumes. It'll keep the codebase less cluttered as non-volumes wouldn't need to be grouped with volumes.
As for size, I doubt anyone minds a few extra megabytes.
It would be great to have a tool for regular volume backups and database dumps. I also support @altendorfme's suggestion.
I would rather we support pre and post backup hooks, and then we can script any additional functionality we need using bash.
- pre backup hooks
- post backup hooks
- on failure hooks
- on warning hooks
- on success hooks
I have a push based monitor for Uptime Kuma and like sending a curl request on successful backup. Uptime Kuma sounds alarms if a successful backup isn’t sent as planned.
Being able to script with these hooks will allow us to handle anything and everything without muddying up the codebase of zerobyte itself.
I created issue #64 as this is technically a separate feature request.
From what I understand, creating a pre/post backup hook that executes commands/interacts with the host OS isn't going to be easy since this container is dockerized. (look at #49)
Either ways, I'd rather have Zerobyte deal with the database dumping part too instead of having everyone create their own script do that.
ZeroByte has a very user-friendly and commendable interface, even though we're talking among more technical people. I don't like the idea of leaving scripts running in external hooks to perform sub-functions. I prefer to trust the tool and its developers ;)
From what I understand, creating a pre/post backup hook that executes commands/interacts with the host OS isn't going to be easy since this container is dockerized. (look at #49)
It can be accomplished by including the docker cli and mounting the docker socket to dump other containers. I do agree though that this adds both complexity and risk due to requiring access to the docker socket.
It can be accomplished by including the docker cli and mounting the docker socket to dump other containers. I do agree though that this adds both complexity and risk due to requiring access to the docker socket.
Sure, it's doable; but specifically in the context of database backups I'd rather have Zerobyte handle it.
This should be requested separately in a different thread.
This should be requested separately in a different thread.
If you read my original suggestion, I did acknowledge that it should be a separate feature request and I linked to a new issue for it. :)
Generally speaking it would be a great feature to have. Most of the database images have the tools included to create and restore backups/dumps. Maybe something like this workflow is possible?
- Zerobyte runs the backup command inside the target database container.
- The backup/dump is written into a backup volume mounted to the database container. Maybe the backup files are overwritten by each backup command execution so deduplication works?
- Zerobyte backups the whole backup volume.
A restore I imagine like this:
- Zerobyte restores the volume at the desired point in time.
- Zerobyte runs the restore command inside the target database container.
What I don't know is in what way we can enable Zerobyte to run commands in arbitrary containers. Probably it's enough to mount the docker.sock and docker exec -it ...?
Furthermore it would be great to have some recipes book inside Zerobyte to manage the backup and restore commands.
Overall I think this complexifies the code too much while the user could simply run dump executions from a script on the host and backup that folder with Zerobyte.
That's what i would prefer. I like the simplicity of Zerobyte, have files, have a cloud account, backup them there, easy. I tend to search for newer tools when older ones get too complicated because the new ones have less feature, but have the core important features. Backups are complicated for me, to make sure they are done correctly, this made it easy, please keep it noob friendly.
@nicotsx Am I free to start this idea again? 😅
I think the tool is not mature enough yet. I have still some work to do before reaching a stable release and maybe it's best to avoid too many features until then