Official Docker image?

[DarrienG]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Issue

System information. For client/server mode post info for both machines.

N/A

Your borg version (borg -V).

N/A

Operating system (distribution) and version.

N/A

Hardware / network configuration, and filesystems used.

N/A

There are a couple of unofficial Docker images for borg, but there's interest in an official Docker image. Especially after the runc vulnerability that was reported the other day, an officially supported image would inspire more trust in it.

I know I could always build one of my own, but the world is moving towards containerized services, and it would likely drive up usage.

I don't mind making a Docker related PR if desired :)

[ThomasWaldmann]

I personally don't use docker.

For which of the supported platforms would you like to add docker images? What exactly would such an image offer?

[DarrienG]

Docker runs on most platforms, so any OS is fine.

Most "single binary" images tend to be based on alpine linux, but Debian, Ubuntu, Fedora, or the like is a fine base image, I don't think anyone would care much.

We're thinking about using Borg at big company XYZ and we have a huge Kubernetes cluster. We would like to use Borg for automating backups and all that jazz.

The image would be used as a sidecar container to run alongside another container in our cluster, and back up any contents that are important.

Containers are supposed to be single purpose, single process, so installing a cron and the borg binary alongside say our Jenkins container isn't much of an option.

As I said before, we can make our own, but an image with official support that is updated as borg is updated, and changed as borg has different dependencies would be great.

[pierreozoux]

I have 2 comments here :) I came to open the same issue ;)

Yes, an official image would be amazing, I already created this one and I'd be happy to PR upstream if it is something you want to have. (It does a bit more than just borg, but well.. we can figure that out during the PR)

We also have 2 docker-compose that goes with that: https://github.com/libresh/compose-borg-server https://github.com/libresh/compose-borg-client

@DarrienG for kubernetes you are actually looking at https://appscode.com/products/stash/ ;) I know it is based on restic, but it is so cool ;) (And I'd love to have that with borg, restic doesn't have compression)

[ThomasWaldmann]

Considering the linux fat binaries are built on debian, debian 64bit would be a good base. In master branch we use stretch currently. Have a look at the Vagrantfile "stretch64" machine.

If this shall have a chance to become part of release process, it must be fully automated.

Also, there is a unfinished ticket of specifying a standard client/server setup, so guess the docker images could be based on that. #1553

[DarrienG]

At work we use a modified version of docker-library, which is likely overkill.

It looks like the docker foundation has their own GitHub integration which will probably do this in just a few clicks.

Set it up to only build when a new tag is added, and you should be good to go.

[ThomasWaldmann]

I'ld like to build binaries on machines / OS I control, not in the "cloud".

[DarrienG]

If you're ok with Travis (which isn't on your machine, but it does look like your build system) you can build through that on a tag release: https://docs.travis-ci.com/user/docker/ and https://docs.travis-ci.com/user/deployment/releases/

Otherwise you'll want to add a webhook that fires a POST or whatever to a server you own.

Building the image will look like this:

# With an authenticated docker client, see https://docs.docker.com/v17.12/docker-cloud/builds/push-images/
LATEST="borg_collective/borg-backup:latest"
DOCKER_IMAGE="borg_collective/borg-backup:$GIT_TAG_FROM_WEBHOOK"

git clone https://github.com/borgbackup/borg
cd borgbackup
git checkout $GIT_TAG

docker build -t "$DOCKER_IMAGE" .
docker tag "$DOCKER_IMAGE" "$LATEST"

docker push "$DOCKER_IMAGE"
docker push "$LATEST"

[DarrienG]

FWIW - I'm not entirely certain how the release process works for borgbackup. If you on the other hand make tags on your local machine and then push the tag up, you could make a hook on your git client itself which will in turn run the above commands quietly in the background.

https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

[pierreozoux]

Where is the pipeline to push to pip? The easiest is really to let https://hub.docker.com/ do the build and host the docker image.

Do you have some infrastructure to host a docker registry and run a pipeline?

I do plan to build one here if you need. (there is no runner nor registry yet, but I'll work on that on the coming weeks).

[ThomasWaldmann]

The stuff needs to be build in the vagrant VM, see Vagrantfile "stretch64" (as I already said above).

No travis, no github, no AWS, ...

I'ld accept a PR that is tested and modifies the Vagrantfile (master branch) so it builds 2(?) docker images.

But as I don't use / don't need docker, I won't implement this myself.

[pierreozoux]

Ok, I understand, I took a look at the vagrant, it will be easy. For the 2 docker image, do you mean:

• a simple one with just Borg bin
• a complexe one with my client/server code to implement #1553

This would be cool indeed and cover most of what people need :)

I'll modify my PR later with that.

@DarrienG you'll have to help me maintainer that, if we share the work, it should be easier :)

@ThomasWaldmann I have more questions:

How do you push to pip? After reading the vagrant file, it was not clear to me. In the Dockerfile, should I use the local bin, or the pip version (like in the PR) (in the second case, I have to make sure the build happens after the pip push). Is it ok for you to host the docker image on hub.docker.com or would you prefer your own registry? In the complexe Docker image, is it ok to ship Prometheus metrics like I currently do, or should I separate this logic? (I think separating would make more sense, but well, up to you :) )

Thanks for your time!

[ThomasWaldmann]

I thought you wanted to make one for the client and one for the server. But just do what makes sense. :)

The source dist (sdist, then uploaded to pypi after signing) currently comes from my development machine, the fat binaries are built and tested in the (self-installed, if possible) vagrant VMs you see in the Vagrantfile and fetched from there to dev machine via vagrant scp. I'ld fetch the docker image from there in the same way.

Inside the docker image, I guess working from source (via the sdist, python setup.py sdist, no need to wait for pypi or downloading it from there) is better than using the fat binary (which is a rather huge wrapper, but that is not needed as that is solved by docker, AFAIK).

I don't know any docker stuff, but an obvious requirement is that users can verify authenticity of any downloaded file via a gpg signature before running it as root and letting it access all their files. Another requirement is not "polluting" the vagrant machines with low-trust stuff (e.g. binary downloads from 3rd party sources, not debian).

Prometheus stuff sounds rather specific, so I guess without makes more sense for broad use.

[DarrienG]

@pierreozoux Happy to help get this going and keep it up to snuff. We're using a company image in some alpha services right now and would definitely like official support. Since you already have a Dockerfile and all that, feel free to add me to the review you make.

I personally don't think Prometheus metrics should be P1/built in.

[DannyBen]

I am also very much in favor of an official image with the borg binary installed.

Seeing the discussion veered a little off course in my opinion, I think that:

1. This should be something simple. No need for any of the dependencies that were mentioned above.
2. Unless there is a complication with borg on Alpine, this most definitely should be an Alpine based image.
3. Testing of the build should be minimal - as you are not testing borg itself, but just the fact that it was built properly and executes (even borg --version could be a sufficient test).
4. There should be no need for a separate server and client images - in the same way there is only one binary.
5. Instead (of 4), the README of this Docker image should simply include instructions (docker run ...) on how to start a server and how to start a client - including all volumes needed etc.
6. I believe it should be acceptable (and even desired) to use a completely separate repository for just this purpose.
7. Tagging of such image on DockerHub should include the borg version.

I also might be able to assist in that area if it is desired. I am a heavy docker user, but unfortunately, a borg noob....

EDIT

Is there any reason not to simply rely on the official alpine package?

FROM alpine
RUN apk --no-cache add borgbackup
ENTRYPOINT ["borg"]
CMD []

Then, all that is left to do is create the README, perhaps add a couple more directives, configure automated build and be done with it. No?

[ThomasWaldmann]

See #4538 with some alpine msgpack issue. Guess until that is solved and also borgbackup updated to latest stable, this is no good base.

[DannyBen]

Personally, I think we can proceed and make preparations for an official docker image, based on the assumption that the alpine package will be fixed soon.

For reference, here is the issue tracker for the msgpack warning: https://bugs.alpinelinux.org/issues/10401

I intend to prepare such an image for my own use, and to follow the principles I mentioned earlier. If anyone might be interested in putting this under borg's official umbrella, I am happy to do so and contribute what I can.

[DannyBen]

If anyone is interested:

1. I have created my take on a borg client image.
2. I also have a version of the server, but I have decided to split these guys for now, since the server requires sshd, and slightly different examples - so I figured I should probably keep these concerns separate.
3. I hope I mounted all of the folders needed for proper operation of borg from the client side (e.g. cache, config, private key)

So, I am putting this here for a couple of reasons:

1. If anyone wants to use it / contribute to it - by all means.
2. If anyone from the borg team wants to use it as a base for an official image - also by all means.

In any case, I hope an official image comes out, and I hope it is not a lot more complicated than this Dockerfile.

[funkyfuture]

I hope it is not a lot more complicated than …

i agree, the use-cases are very different, so a base image should really just provide the software and its dependencies, but not even define an ENTRYPOINT or CMD.

[euh2]

I very much like @DannyBen take on this.

For quite some time I used the following line in my Dockerfile to have the latest Borg version (should be extended by some tests):

RUN curl --silent "https://api.github.com/repos/borgbackup/borg/releases/latest" | jq -r '.assets[] | select(.name | endswith("linux64")).browser_download_url' | wget -nv --show-progress --progress=bar:force:noscroll -i - \
    && mv borg* /usr/local/bin/borg && chown root:root /usr/local/bin/borg && chmod 755 /usr/local/bin/borg \
    && /usr/local/bin/borg -V

[staticvoidmaine]

Not sure if there's been any more movement on this, but FWIW I also agree with @DannyBen and @funkyfuture in that the base image should be generic and not preclude some people from using it due to it being opinionated.

Additional functionality and/or opinions on its use should be specified in people's own images derived from the official base image.

[alexxxnf]

Hi guys,

This issue has not been updated for a quite a while. What's the current status?

[ThomasWaldmann]

There's PR #4372 open since quite a while.

While I don't personally use docker and thus have no clue about it, I feel it is too complex and should be way simpler (IIRC there were also some comments along these lines).

There is also the maintenance issue: if we merge some "official Docker support", we need to maintain it. I obviously won't do it and so there is some fear that the more complex it is, the less maintainable it will be.

So, blockers: a simple thing and maintainers for it.

[EthraZa]

Why are people making it complex? You just need to choose a official base image (ie. debian:stable-slim) and add something as simple as one line to the Dockerfile to add the borg binary and that's it, everything else is aleready there.

Example:

FROM debian:stable-slim
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get -y update && apt-get -y install --no-install-recommends \
        fuse \
        openssh-server \
        wget && \
        apt-get -y clean && \
        apt-get -y autoclean && \
        apt-get -y autoremove && \
        rm -rf /var/lib/apt/lists/*

RUN groupadd -g 1000 borg && \
        groupadd -g 500 core && \
        groupadd -g 233 docker

RUN useradd -m -u 1000 -s /bin/bash -c "Stevedore" -g borg borg -G adm,backup,core,disk,docker,mail,nogroup,operator,root,staff,sudo,tape,tty,users,www-data

RUN wget https://github.com/borgbackup/borg/releases/latest/download/borg-linux64 -O /usr/bin/borg && \
        chmod 755 /usr/bin/borg

ENTRYPOINT /usr/bin/borg

Did not tested this Dockerfile, but if it's not this, it's very close to this. Borg already use env variables that can be passed by environment flags in Docker, but, anyways, it's just borg usage, not exactly Docker specific.

[EthraZa]

Done, Just made my own Backup oriented Docker image:

https://github.com/EthraZa/BackupOps

[yash-fn]

Very nice thank you!

[hatl]

Maybe this can serve as the "official" image? https://github.com/borgmatic-collective/docker-borgmatic

[stuaxo]

I had a look at the Alpine related issues above, and they have all been fixed, so having an Alpine based docker image should not be an issue any more.

[azlux]

I also have my own since there are no official ones with borg and tool for database backup. (https://github.com/azlux/borgbackup-docker) There are many way to create a image. Do we want cron, tool to export, borgmatic, how many env variables, DB backup before borg... Maybe having 2-3 images for differences used. Or just one with the borgbackup tool only.

[RonnyPfannschmidt]

A single image whose entrypoint is a dumb init wrapped call to borg

Stuff like cron in a container is a no go

[enpaul]

Hi all, I'm new to Borg but have a lot of experience with Docker and am looking at bringing Borg into my container infrastructure. If people are still looking for a solution here, I'd be happy to look at resurrecting @pierreozoux 's PR (#4372) for an official image. I can also commit to maintaining the container in the future.

@ThomasWaldmann are there docs somewhere on the release process I could read so I can make sure the docker implementation can integrate with your existing process?