swarm-cronjob
swarm-cronjob copied to clipboard
docker cap_add seems to work only with replica to 1 and not in scheduled mode
Hi, and thanks for your work !
We use cronjob to start jobs in our clusters and for some reasons we need for one of our container to set the mac_address (licensing stuff). We found that adding cap_add: - NET_ADMIN allow us to override that value and we are ok with that.
We then tried to deploy such a container in our swarm and found that :
- if the container is started with deploy: replica: 1, it is running well
- if the container is started by the scheduler : swarm.cronjob.schedule=* * * * * for example, we see the error : ip: ioctl 0x8914 failed: Operation not permitted
In the docker documentation, we can see that : `Note when using docker stack deploy
The cap_add and cap_drop options are ignored when deploying a stack in swarm mode`
But it seems that with our used version of docker (20.10.10) and cronjob (1.10.0, latest) the cap_add is NOT ignored, or at least for the replica :1 option.
Here is an example of our iml file :
version: '3.8'
services:
stream:
image: alpine
entrypoint: [ "/bin/sh","-c" ]
command: >
"ip link set dev eth0 down
&& ip link set dev eth0 address fa:16:3e:87:02:d7
&& ip link set dev eth0 up"
deploy:
replicas: 0
labels:
- "swarm.cronjob.enable=true"
- "swarm.cronjob.schedule=* * * * *"
- "swarm.cronjob.skip-running=true"
cap_add:
- NET_ADMIN
Apparently capabilities are not handled for swarm services. In #203 I have added a new label swarm.cronjob.capabilities
where we can set kernel capabilities when the service is updated but does not seem to work:
# cap.yml
version: "3.8"
services:
test:
image: alpine:edge
command: >
/bin/sh -c "apk add libcap-utils && capsh --print | grep Current: | cut -d' ' -f2"
deploy:
replicas: 0
labels:
- "swarm.cronjob.enable=true"
- "swarm.cronjob.schedule=*/5 * * * * *"
- "swarm.cronjob.skip-running=true"
- "swarm.cronjob.capabilities=NET_ADMIN"
cap_add:
- NET_ADMIN
$ docker service logs cap_test
...
cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep
Also looking at the docs it seems it's not supported when deploying a stack in swarm mode: https://docs.docker.com/compose/compose-file/compose-file-v3/#cap_add-cap_drop
I'm not sure why it works with replicas: 1
though.
Thanks for your reply :) We managed to use a different solution (not requiring NET_ADMIN) but it may be interesting to be able to use it anyways.
You said that even with the dedicated label it still does not work; it is because of swarm ? Do you think it is a "bug" that it works with replicas: 1 ?