Create Kubernetes specific install docs

[loshz]

I recently started migrating from a competitor CI to Buildkite and so far I am loving the service! I am running my own agents in a self-hosted Kubernetes cluster with no problems so far.

Even though I am running on AWS, I loosely followed the GCloud docs and got things set up fairly quickly. However, I have a couple of "concerns" about using the approach the docs recommend.

Firstly, from my experience, it is not recommended to mount the host Docker socket (Docker-on-Docker) into a pod as this could potentially expose the following security issues:

• Malicious code on the agent could have full access to the Docker daemon on the host machine.
• Any Docker resources created via an agent build would be unknown to the kubelet and could cause scheduling/resource issues.
• Running the buildkite-agent with a privileged security context could again mean code ran via a build could potentially have root access on the host machine.

I am by no means a security expert, but after using Kubernetes in production for several years I have come to learn that some of the above practices could be dangerous.

That being said, I would like to propose a new, Kubernetes specific, set of installation docs- and I would be more than happy to hep write them alongside anyone else!

I have the following Docker-in-Docker (dind) setup running in one of my clusters:

spec:
  containers:
    - name: dind
      image: 'docker:18.06-dind' # must be the same version of Docker as the host machine
      securityContext:
        privileged: true
      volumeMounts:
        - name: dind-storage
          mountPath: /var/lib/docker
    - name: buildkite-agent
      image: 'buildkite/agent:3.22.0'
      livenessProbe:
        httpGet:
          path: /
          port: 6000
        initialDelaySeconds: 30
        periodSeconds: 3
      env:
        - name: DOCKER_HOST
          value: 'tcp://localhost:2375'
        - name: BUILDKITE_AGENT_HEALTH_CHECK_ADDR
          value: ':6000'
        - name: BUILDKITE_AGENT_TOKEN
          valueFrom:
            secretKeyRef:
              name: buildkite-agent
              key: token
  volumes:
    - name: dind-storage
      emptyDir: {}

Happy to provide any further information!