Fail when update docker package

[nickvth]

Version 0.6.1

Platform/Architecture Ubuntu

Describe the bug job will fail when updating docker package

To Reproduce apt-get update apt-get upgrade -y "docker-ce update avaliable"

Expected behavior Update docker package without crashing

Actual behavior Docker daemon crash

Additional context Jun 23 14:41:21 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:21.552045334Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Jun 23 14:41:21 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:21.559750482Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.258549430Z" level=info msg="Container 1dad5e30dd6e75b2e8c5d9dcd89a064db8ebf6186b6753e95dd5bf9d3a801fed failed to exit within 10 seconds of signal 15 - using the force" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.259538935Z" level=info msg="Container cc195360ad0c57786b0e0bd7e1d9a5e5b3e2121da6ce2b4cfcc846dd1e0704a3 failed to exit within 10 seconds of signal 15 - using the force" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.267219221Z" level=info msg="Container b188d2811e8ab6ff7375fd932a255571c804c5b29341e6e106d965a57e0b7fd7 failed to exit within 10 seconds of signal 15 - using the force" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.351366864Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.397157353Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.495799768Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.549875577Z" level=info msg="stopping event stream following graceful shutdown" error="" module=libcontainerd namespace=moby Jun 23 14:41:31 worker-dcg-01 dockerd[1593]: time="2020-06-23T14:41:31.550094969Z" level=info msg="Daemon shutdown complete"

[dweomer]

@nickvth I don't understand this part (I assume it is a cut-and-paste error):

apt-get upgrade -y "docker-ce update avaliable"

When I try that command locally (using sudo) on my Ubuntu Focal workstation I get:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package docker-ce update avaliable

---

That aside, can you explain what you mean by the docker daemon "crashes" ? Are you able to start it up manually using systemctl?

Also, am I to infer that you are running your k8s installation backed by Docker via the dockershim? This is a use-case that we have not tested at all, I must confess.

[nickvth]

@dweomer

• We installed docker-ce from docker community edition repository
• Yes our installation is running RKE + docker-ce on ubuntu 18.04
• If Plan --> Job --> Pod is running with docker-ce package upgrade, docker daemon crashed and no packages are upgraded.
• After login with ssh, I can start docker with systemctl, but dpkg is broken and dpkg --configure -a command is needed to fix.
• When i hold package docker-ce apt-mark hold docker-ce the upgrade is successful, but we want to upgrade the docker package (CVE etc..) as well
• Manualy upgrade docker-ce package is working

Plan

---
apiVersion: v1
kind: Secret
metadata:
  name: bionic-workers
  namespace: system-upgrade
type: Opaque
stringData:
  upgrade.sh: |
    #!/bin/sh
    set -e
    export DEBIAN_FRONTEND=noninteractive
    secrets=$(dirname $0)
    apt-get update
    apt-get upgrade
    if [ -f /run/reboot-required ]; then
      cat /run/reboot-required
      reboot
    fi
---
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: bionic-workers
  namespace: system-upgrade
spec:
  concurrency: 1
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/worker, operator: Exists}
  serviceAccountName: system-upgrade
  secrets:
    - name: bionic
      path: /host/run/system-upgrade/secrets/bionic
  drain:
    force: true
  version: bionic
  upgrade:
    image: ubuntu
    command: ["chroot", "/host"]
    args: ["sh", "/run/system-upgrade/secrets/bionic/upgrade.sh"]