script.sh: 2: docker: not found

[uralsemih]

I deployed Jenkins on Kubernetes and it is up and running without any issue. But when I would like to run docker command in the pipeline I am getting and docker:not found error. What should I change in the helm chart to be able to run docker commands?

node {
  stage('SCM') {
    checkout(scm)
  }
  stage('Build') {
    echo 'Building Project'
    sh """
      docker pull alpine
    """
  }

----
+ docker pull alpine
/home/jenkins/agent/workspace/experiments-pipeline@tmp/durable-4c43b96b/script.sh: 2: docker: not found

Here is values.yaml agent: enabled: true volumes:

• type: HostPath hostPath: /Users/username/workspace mountPath: /Users/username/workspace
• type: HostPath hostPath: /var/run/docker.sock mountPath: /var/run/docker.sock persistence: enabled: true storageClass: annotations: {} labels: {} accessMode: "ReadWriteOnce" size: "8Gi" volumes:
  • name: source-code hostPath: path: /Users/username/workspace mounts:
  • mountPath: /Users/username/workspace name: source-code

Version of Helm and Kubernetes

- Helm: v3.8.0
- Kubernetes: v1.23

Chart version

jenkins -3.11.4

What happened?

No response

What you expected to happen?

able to run docker command in the pipeline

How to reproduce it

No response

Anything else we need to know?

No response

[torstenwalter]

docker is not included in the default build pod (agent image).

[uralsemih]

@torstenwalter so do i have to use a custom image likewise below ? Will it break something to use custom image or is there any other suggested way ?

FROM jenkins/inbound-agent:4.11.2-4
USER root
RUN set -eux && \
    apt-get update && \
    apt-get install -y docker.io docker-compose
RUN docker --version
USER jenkins

[torstenwalter]

It's possible to build a custom image and install docker there.

I personally prefer container composition. So just using the jnlp container with the inbound-agent image to connect to the Jenkins controller and run one or more additional images within that pod as sidecars which contain the software I need e.g. docker.

Within the pipeline it looks something like this:

pipeline {
    agent {
        kubernetes {
            defaultContainer 'my-container'
            yamlFile 'buildpod.yaml'
        }
    ...

I specify my build pod in a buildpod.yaml within the same repository. It's possible to inline it as well but having it separate gives me normal syntax highlighting etc.

In this yaml file I only specify additional images which I want to use. So I do not specify a container with the name jnlp. The plugin automatically injects the jnlp container in that case for me and I do not have to worry about it.

If you want to execute code in a specific container you can use the container(...) syntax:

stage('test') {
  steps {
    container('golang') {
       sh 'go test ./...'
    }
  }
}

If you omit the container block the code will be executed in the default container, which I specified above to be my-container specified with defaultContainer 'my-container'. If you don't do that the default will be the jnlp container which most likely does not contain the tools you want (otherwise you would not make use of sidecars).

Here is an example of how such a buildpod.yaml could look like:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: golang
      image: golang:1.17
      command:
        - cat
      tty: true
      resources:
        requests:
          memory: 3Gi
          cpu: "2"
        limits:
          memory: 5Gi
      imagePullPolicy: Always
    - name: my-image
      image: registry/image:1.0.0
      command:
        - cat
      tty: true
      env:
        - name: DOCKER_HOST
          value: tcp://localhost:2376
        - name: DOCKER_TLS_VERIFY
          value: "1"
        - name: DOCKER_CERT_PATH
          value: /certs/client
      volumeMounts:
        - name: docker-client-certs
          mountPath: /certs/client
      resources:
        requests:
          memory: 256Mi
          cpu: 50m
        limits:
          memory: 512Mi
      imagePullPolicy: Always
    - name: dind
      image: docker:19.03.6-dind
      env:
        - name: DOCKER_TLS_CERTDIR
          value: /certs
      volumeMounts:
        - name: docker-client-certs
          mountPath: /certs/client
        - name: docker-config
          mountPath: /etc/docker/daemon.json
          subPath: daemon.json
      securityContext:
        privileged: true
      resources:
        requests:
          memory: 1Gi
          cpu: "1"
        limits:
          memory: 2Gi
      imagePullPolicy: Always
  volumes:
    - name: docker-client-certs
      emptyDir:
        medium: Memory
    - name: docker-config
      configMap:
        name: docker-config

Here I specified three containers:

• golang: the official golang image, which makes it easy to switch versions
• my-image: that's the one with the docker cli and other tools I use in my pipeline)
• dind: this image starts the docker daemon

I am sharing docker client secrets between the image running the docker daemon and the one which contains the docker cli and other tools.

Please also note that all images shown here run a long running process which does not exit. If you don't do that Kubernetes will restart the pod all the time as it thinks it's unhealthy. In this example I just execute cat as long running process. That's also the reason why I set tty: true.

This example may not be production ready and it has some flaws e.g. running as privileged but it shows the concept of container composition for build pods.

For completeness also the docker-config ConfigMap which is used to configure the docker daemon:

apiVersion: v1
kind: ConfigMap
metadata:
  name: docker-config
data:
  daemon.json: |
    {
      "bip": "192.100.1.5/24",
      "fixed-cidr": "192.100.1.5/25",
      "fixed-cidr-v6": "2001:db8::/64"
    }

[uralsemih]

@torstenwalter thanks for the detailed information but this is a bit complex for the local enviroment.

Actually, I was able to install docker via a custom image as below but this time I am getting a permission error where I am stuck. Added Jenkins user to docker group did not fix permission issue. In the values.yaml. I tried to add command: ["sh", "-c", "chown -R jenkins:docker /var/run/docker.sock"] but this time pod CrashLoopBackOff wit below error

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$2 (file:/var/jenkins_cache/war/WEB-INF/lib/guice-4.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-02-11 11:59:28.658+0000 [id=29]	INFO	jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
2022-02-11 11:59:28.951+0000 [id=31]	INFO	jenkins.InitReactorRunner$1#onAttained: Started all plugins
2022-02-11 11:59:29.152+0000 [id=31]	INFO	jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
2022-02-11 11:59:36.001+0000 [id=32]	INFO	jenkins.InitReactorRunner$1#onAttained: System config loaded
2022-02-11 11:59:38.100+0000 [id=32]	WARNING	i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.csanchez.jenkins.plugins.kubernetes.PodTemplate#listener: type is abstract but not Describable.
2022-02-11 11:59:38.229+0000 [id=32]	SEVERE	jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init
io.jenkins.plugins.casc.ConfiguratorException: Item isn't a Scalar
	at io.jenkins.plugins.casc.model.CNode.asScalar(CNode.java:26)
	at io.jenkins.plugins.casc.impl.configurators.PrimitiveConfigurator.configure(PrimitiveConfigurator.java:44)
	at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:351)

agent:
  enabled: true
  image: "jenkinsinbound-agent"
  tag: "4.11.2-5"
  workingDir: "/home/jenkins/agent"
  imagePullSecretName:
  componentName: "jenkins-agent"
  websocket: false
  privileged: false
  runAsUser: 
  runAsGroup:
  volumes:
  - type: HostPath
    hostPath: /Users/uralsem/workspace
    mountPath: /Users/uralsem/workspace
  - type: HostPath
    hostPath: /var/run/docker.sock
    mountPath: /var/run/docker.sock
   command: ["sh", "-c", "chown -R jenkins:docker /var/run/docker.sock"]

FROM jenkins/inbound-agent:4.11.2-4
USER root
RUN set -eux && \
    apt-get update && \
    apt-get install -y docker.io docker-compose && \
    curl -sS https://raw.githubusercontent.com/HariSekhon/bash-tools/master/clean_caches.sh | sh
RUN usermod -g docker jenkins
RUN usermod -a -G jenkins jenkins
USER jenkins

+ docker pull alpine
Using default tag: latest
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/create?fromImage=alpine&tag=latest": dial unix /var/run/docker.sock: connect: permission denied

[torstenwalter]

Your chown commands exits immediately. No matter if it's failing or succesful once the process is finished the container will be restarted by Kubernetes.

[uralsemih]

so what is the best way of granting permission through the chart for /var/run/docker.sock

[torstenwalter]

I never tried as it's from a security point of view a bad idea.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[aushafy]

I have run Jenkins in K8s for more than 6 months, I have another suggest for you to use different approach for Building Docker Image inside of Kubernetes, you can use Docker-in-Docker or Kaniko for more secure rather than Install Docker inside of inbound-agent. feel free to contact me if you want more details

[gwasky]

https://blog.thecloudside.com/docker-in-docker-with-jenkins-pod-on-kubernetes-f2b9877936f2

[martmalo]

I have run Jenkins in K8s for more than 6 months, I have another suggest for you to use different approach for Building Docker Image inside of Kubernetes, you can use Docker-in-Docker or Kaniko for more secure rather than Install Docker inside of inbound-agent. feel free to contact me if you want more details

@aushafy can you contact me about this issue on [email protected]?

[SheplX]

so what is the best way of granting permission through the chart for /var/run/docker.sock

hi, try this:

FROM jenkins/inbound-agent USER root RUN set -eux &&
apt-get update &&
apt-get install -y docker.io docker-compose &&
curl -sS https://raw.githubusercontent.com/HariSekhon/bash-tools/master/clean_caches.sh | sh ENV USER jenkins RUN usermod -aG docker ${USER} RUN newgrp docker

[stevelaclasse]

Here are the settings that i have used to make it work. I have unfortunately used the root user for the agent .

Jenkins agent custom Dockerfile:

FROM jenkins/inbound-agent:3107.v665000b_51092-15
USER root
RUN apt update && apt install apt-transport-https ca-certificates curl gnupg lsb-release -y
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
    $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update && apt -y install docker-ce docker-ce-cli containerd.io 

RUN usermod -aG docker jenkins

USER jenkins 

Jenkins agent helm values custom file:

agent:
  image: "#custom-registry-with-new-jenkins-agent/jenkins-agent"
  tag: "latest"
  #set the user to root to access the share workspace on the host, as Jenkins create the PVC with root access
  #didn't work with Userid 1000
  runAsUser: 0
  fsGroup: 0
  volumes:
  - type: HostPath
    hostPath: /tmp/jenkins_workspace
    mountPath: /tmp/jenkins_workspace
  - type: HostPath
    hostPath: /var/run/docker.sock
    mountPath: /var/run/docker.sock
  resources:
    requests:
      cpu: "512m"
      memory: "2048Mi"
    limits:
      cpu: "512m"
      memory: "2048Mi"
  workingDir: /tmp/jenkins_workspace