docker-ssh-agent
docker-ssh-agent copied to clipboard
jenkinsci
Remove VOLUME from Dockerfiles or support changing uid/gid of jenkins user
What feature do you want to see added?
Would be nice if Dockerfiles do not use VOLUME instruction. Problem is that then the directory is only modifiable with buildkit. For example without builtkit it is not possible to assign a different uid to jenkins user. This would allow to match already existing users on host system.
Sure the problem with builtkit does not exists. But builtkit is not documented while using direct api calls, e.g. through go docker sdk: https://pkg.go.dev/github.com/docker/docker/client#Client.ImageBuild
Is there any benefit in using "VOLUME" directly in Dockerfile? Is there a difference between an anonymous volume and using directly docker file system? If someone needs a volume he/she could provide it while starting the docker container.
Some background: https://stackoverflow.com/a/55516433
#export DOCKER_BUILDKIT=1
docker build -ttest - <<EOF
FROM jenkins/ssh-agent:4.3.0-jdk11
RUN usermod -u 1200 jenkins; ls -ldn /home/jenkins >/test.txt
EOF
docker run --rm -it --entrypoint= test bash -c 'cat /test.txt;ls -ldn /home/jenkins'
Output:
drwxr-xr-x 2 1200 1000 4096 Jun 3 16:10 /home/jenkins
drwxr-xr-x 2 1000 1000 4096 Jun 3 16:28 /home/jenkins
The first line is during build and the second line is using built image. As you can see the change to uid 1200 is lost.
With enabled builtkit the result would be fine (but as explained difficult to archive this docker api calls):
drwxr-xr-x 1 1200 1000 4096 Feb 19 16:25 /home/jenkins
drwxr-xr-x 2 1200 1000 4096 Jun 3 16:29 /home/jenkins
As a workaround I copy/change in Dockerfile home directory to a different directory but this does not feel nice.
Upstream changes
No response
Hi @waschik , thanks for opening this issue!
At high level (and as a general practise as underlined in https://docs.docker.com/storage/volumes/), the volume on a container image allows to have close-to-native I/O performances, along with persistence and extensibility.
In the case of the Jenkins agents, using the VOLUME directory for the jenkins home allows the best performance to have the data read and written in the directory (usually the default agent workspace, but also all the build cachings such as ~/.m2 for instance in maven world).
Using the default container root system would result in poor performances and non optimal defaults.
It would also break any installation using the flag --read-only of docker run (or any equivalent) that mounts the root filesystem on read-only to prevent any processus breakout.
I assume that you have a legit issue that could lead to either an improvement on the default entrypoint or at least in an improved documentation: can you explain the problem that you are trying to solve at higher level?
I understand that you want to use another UID than the default provided: what is the constraint behind this?
Besides, is there a reason to not reuse the Dockerfile in this system and change the default BUILD_ARG specifying the UID, so you would have you own version of it?
In term of "implementation", is there a reason to keep the username jenkins for the new user with new UID that you are creating? That would help creating another user than the default, so you would control the whole thing. What do you think?
Okay. I thought most will mount the volume anyway during docker start. Then volume inside Dockerfile would be optional.
But most usecases will be different. Then I will live with renaming home directory (or add another user) as workaround. Alternatively I could copy Dockerfile and change it accordingly. But the later I normally avoid because it needs more attentions for updates. Then I guess you will close the issue with something "Won't fix".
Background is a jenkins instance which should be able to run docker containers inside jenkins and modify data on an external mount which might be only writeable by user with an uid different then 1000. So I write a Dockerfile which adds the user to group "docker" (same gid as host) and also modify uid to the same has host. (If you ask for security, the host does not run something else then jenkins and application to start jobs).
I might have also no internet on the machines running jenkins-ssh and do not know the uids in advance. So I create tar files of the docker images and move these to target file and use another Dockerfile to change there uid (plus adding docker group).
Thanks for giving the details.
Had a similar case a few years ago and I can share the solution with you. Do you mind opening a topic on community.jenkins.io that points to this issue, and where you re-state the use case you just described? Don't forget to mention me (same username as in GitHub) so i'll be able to share it as soon as it is created.
Then, I'll close the issue here, as the initial request ("remove volume directive") won't be accepted, as you wrote :)
However, depending on how the discussion goes on community.jenkins.io, if you succeed in this use case, it could be interesting to propose a blog post on jenkins.io, or a documentation guide, to share your use case, what do you think?
Hi @waschik , can you post a link to the discussion ? I am also interested in the solution.
Thanks,
@ieugen I have no link to another discussion except this. For the problem, as mentioned, I changed additionally to the uid also the home directory /home/jenkins to a different one (option -d for usermod) and then it works. For me it is easier to use an image based on the image on dockerhub then Download the dockfile and modify it and build image myself.
With performance (better/worse) I have not much experience. Most applications I use are cpu bound and not io bound.
This isn't an "enhancement"; it's a BUG. And quite a bad one at that.
- The /home/jenkins directory shouldn't be mounted on a volume. When you do that, and Docker mounts it as root, now you can't SSH into the container because the /home/jenkins/.ssh/authorized_keys file is now owned by root, but you're truing to SSH in as jenkins. Useless.
- The working directory and the home directory of the user shouldn't be inextricably linked.
- The VOLUME keyword is forcing Docker to create temporary volumes for directories which one most probably DOES NOT want on permanent storage, and most probably WILL NOT be using anyway. This results in a bunch of orphaned volumes (three or four per run) each time the container is stopped and restarted. If you don't clean them up, eventually they'll take you out.
Just take the VOLUME line out, release a new image, and be done with it. It's a one-liner. Get it done.
@roncemer please don't hesitate to make a pull request with your proposal.
As a side note, these are my own thoughts, I'm not speaking for the Jenkins community. Please don't require the Jenkins community to do this correction, even if it's a one-liner. I feel this is disrespectful to the developers. Nobody is entitled to support from volunteer FOSS projects, and developers absolutely do deserve respect. Your priorities may not be the same as the priorities of the project, or the time developers have to correct this bug may not be sufficient to correct it (because of higher priority bugs or features).
If you have the solution, feel free to share it with the community, but don't force the community to implement it for you. I share my feelings with you in all friendship. Take care.
Community member hat on
This isn't an "enhancement"; it's a BUG. And quite a bad one at that. (...) Just take the VOLUME line out, release a new image, and be done with it. It's a one-liner. Get it done.
Please watch your tone, we are not entitled to any kind of support: it is an open community with people working on their own time. It's not because you consider this a bug that other do and discussion is here to reach consensus. There is no one forbidding you to open a pull request with explanations (pros, cons, trade offs) and if the process is frustrating you, there is always the solution of maintaining your own Dockerfile and docker image for your own use case.
Maintainer hat on
That being said, both the initial request and your request are legit but there are numerous use cases to take in account.
1. The /home/jenkins directory shouldn't be mounted on a volume. When you do that, and Docker mounts it as root, now you can't SSH into the container because the /home/jenkins/.ssh/authorized_keys file is now owned by root, but you're truing to SSH in as jenkins. Useless.
As per the official Docker documentation (https://docs.docker.com/engine/reference/builder/#volume) the permissions of the mounted volume are set to the permissions from the base image.
The point 2. that you make is valid and would solve that issue differently.
However there is an issue hidden in your setup, underline by this behavior: if the permissions of /home/jenkins/.ssh/authorized_keys are set to the root user, then either you are bind-mounting from the host (which relies on the permissions of the mounted file... on the host) or there was a permissions done in the data volume that created this issue (and that can happen again).
2. The working directory and the home directory of the user shouldn't be inextricably linked.
You are correct: it creates issues and a good compromise would be to:
- Define the agent working directory as a data volume (
/home/jenkins/workdiris the default as far as I remember) - Document that the agent working dir configuration item should be set to this value in Jenkins UI, or at least declaring the non default value as a data volume
- Remove the jenkins user home from the data volume
3. The VOLUME keyword is forcing Docker to create temporary volumes for directories which one most probably DOES NOT want on permanent storage, and most probably WILL NOT be using anyway. This results in a bunch of orphaned volumes (three or four per run) each time the container is stopped and restarted. If you don't clean them up, eventually they'll take you out.
I strongly disagree with this one: the performances of the layered filesystem are terrible and read/write into non-data volumes are recipe for poor user experience.
Yes it creates a lot of volumes but there are multiple ways to cleanup docker volumes (docker system prune --volumes for instance) that are unused.
Also, you can define volumes to either tmpfs systems (https://docs.docker.com/storage/tmpfs/) if you do not want to persist (which make the mounts.... data volumes in Docker API).
We are going to change this behavior because it is a legit request. But we need time and availability. If you disagree with this, then please contribute or fork.
This issue is now fixed with the latest version 4.5.1:
docker build -ttest - <<EOF
FROM jenkins/ssh-agent:4.5.1-jdk11
RUN usermod -u 1200 jenkins; ls -ldn /home/jenkins >/test.txt
EOF
docker run --rm -it --entrypoint= test bash -c 'cat /test.txt;ls -ldn /home/jenkins'
[+] Building 3.5s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 134B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/jenkins/ssh-agent:4.5.1-jdk11 2.8s
=> [1/2] FROM docker.io/jenkins/ssh-agent:4.5.1-jdk11@sha256:e8cb848f04d7539c6493f097dee56561b3bf027b22c3ba78f57e777334dcaf89 0.1s
=> [2/2] RUN usermod -u 1200 jenkins; ls -ldn /home/jenkins >/test.txt 0.4s
=> exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:0877102c701e593b61f374fd267f5ccffc002d72258189326298610872efa564 0.0s
=> => naming to docker.io/library/test 0.0s
drwxr-xr-x 1 1200 1000 4096 Nov 25 11:58 /home/jenkins
drwxr-xr-x 1 1200 1000 4096 Nov 25 11:58 /home/jenkins
In #165 (included in the 4.5.x releases), the jenkins user home was removed from the "VOLUME" directive so you can fine tune it.
However we kept the data volume for the AGENT_WORKDIR and the /home/jenkins/.jenkins because 9/10 of the use cases requires these to be an anonmyous data volume (for performance reasons).
I recommend you to specify another working dir:
docker build -ttest - <<EOF
FROM jenkins/ssh-agent:4.5.1-jdk11
ENV AGENT_WORKDIR=/agent-workdir
RUN usermod -u 1200 jenkins;
RUN chown -R 1200 "${AGENT_WORKDIR}""
EOF
If you are using Docker and are bothered by the anonymous data volumes, it is a common production practise to add the docker system prune --volumes --force command to a crontab.
I'm closing this issue (it was long due). If you still have an issue with the latest release, please reopen and describe the problem you have.
Hope it helped
If you are using Docker and are bothered by the anonymous data volumes, it is a common production practise to add the docker system prune --volumes --force command to a crontab.
It's common production practice to not utilize cron scripts at all. They're explicitly bad practice, in fact.
kept the data volume for the AGENT_WORKDIR and the /home/jenkins/.jenkins because 9/10 of the use cases requires these to be an anonmyous data volume (for performance reasons)
I'd be curious for the specs on what the performance reasons are for these being loaded as anonymous volumes?
If the user desires volumes to be mounted then the onus is on them to mount that volume at container runtime (through the docker run -v flag).
Should a new issue be opened or can this one be reopened to still track removing any VOLUME instruction in the image creation manifest?
EDIT: After reading more of the discussion it looks like the "performance" argument comes from performance of having a data disk for persistent data. But not all use cases are going to want persistent data. And the flexibility is already provided to users that do want it via the docker run --volume flag.
To add to that, there's only a performance benefit discussed in the docker volume documentation relative to bind mounts.
