Got 'could not synchronize with container process' error on 1.10.3-33

[smarterclayton]

Fast terminating process returned unexpected error from docker run. This is an intermittent failure.

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-1.10.3-33.gitee81b72.fc23.x86_64
 Go version:      go1.5.4
 Git commit:      ee81b72/1.10.3
 Built:
 OS/Arch:         linux/amd64

$ docker run -it --entrypoint=/bin/true docker.io/openshift/deployment-example:v1
^CError response from daemon: Cannot start container b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b: [9] System error: could not synchronise with container process

Jul 07 17:34:01 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:01.410350027Z" level=info msg="{Action=attach, ID=b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b, LoginUID unknown, PID unknown}"
Jul 07 17:34:03 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:03.830875842Z" level=info msg="{Action=start, ID=b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b, LoginUID unknown, PID unknown, Config={Hostname=b953ede06307, AttachStdin=true, AttachStdout=true, AttachStderr=true, ExposedPorts=map[8080/tcp:{}], Tty=true, OpenStdin=true, StdinOnce=true, Env=[COLOR=#006e9c], ArgsEscaped=false, Image=docker.io/openshift/deployment-example:v1, Entrypoint={parts:[/bin/true]}, NetworkDisabled=false, Labels=map[], StopSignal=SIGTERM}, HostConfig={LogConfig={Type:json-file Config:map[]}, NetworkMode=default, PortBindings=map[], RestartPolicy={Name:no MaximumRetryCount:0}, DNS=[], DNSOptions=[], DNSSearch=[], Privileged=false, PublishAllPorts=false, ReadonlyRootfs=false, ShmSize=67108864, Resources={CPUShares:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CpusetCpus: CpusetMems: Devices:[] KernelMemory:0 Memory:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:0xc821e83cb0 OomKillDisable:0xc821e83ca0 PidsLimit:0 Ulimits:[]}}}"
Jul 07 17:34:03 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:03.835377384Z" level=info msg="{Action=remove, ID=1ee2a78152499508a1290e5f0d4a1934d0e3f733a1ab82778661744e0f629675, Username=vagrant, LoginUID=1000, PID=14929}"
Jul 07 17:34:03 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:03.961440952Z" level=info msg="Config: &{CommonCommand:{ContainerPid:0 ID:b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b InitPath:/.dockerinit MountLabel:system_u:object_r:svirt_sandbox_file_t:s0:c895,c968 Mounts:[{Source:/var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/secrets Destination:/run/secrets Writable:true Data: Propagation:} {Source:/var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/resolv.conf Destination:/etc/resolv.conf Writable:true Data: Propagation:rprivate} {Source:/var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/hostname Destination:/etc/hostname Writable:true Data: Propagation:rprivate} {Source:/var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/hosts Destination:/etc/hosts Writable:true Data: Propagation:rprivate} {Source:/var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/shm Destination:/dev/shm Writable:true Data: Propagation:rprivate}] Network:0xc821e12330 ProcessConfig:{CommonProcessConfig:{Cmd:{Path: Args:[] Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=b953ede06307 TERM=xterm COLOR=#006e9c] Dir: Stdin:<nil> Stdout:<nil> Stderr:<nil> ExtraFiles:[] SysProcAttr:0xc8231c3680 Process:<nil> ProcessState:<nil> lookPathErr:<nil> finished:false childFiles:[] closeAfterStart:[] closeAfterWait:[] goroutine:[] errch:<nil>} Tty:true Entrypoint:/bin/true Arguments:[] Terminal:<nil>} Privileged:false User: Console:} ProcessLabel:system_u:system_r:svirt_lxc_net_t:s0:c895,c968 Resources:0xc822e85c00 Rootfs:/var/lib/docker/devicemapper/mnt/c880199ffdf5dead64bf1b89a8dfc440a9efe47be27c9260aa401671de352c85/rootfs WorkingDir: TmpDir:/var/lib/docker/tmp/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b204334597 ContainerJSONPath:/var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/config.v2.json} AllowedDevices:[0xc82008ad20 0xc82008ad70 0xc82008adc0 0xc82008ae10 0xc82008ae60 0xc82008aeb0 0xc82008af00 0xc82008af50 0x20a5b40 0x20a5ba0 0x20a5c00 0x20a5c60 0x20a5cc0 0x20a5d20] AppArmorProfile: AutoCreatedDevices:[0xc82008afa0 0x20a5b40 0x20a5ba0 0x20a5c00 0x20a5c60 0x20a5cc0 0x20a5d20] CapAdd:[] CapDrop:[] CgroupParent:system.slice GIDMapping:[] GroupAdd:[] Ipc:0xc822862e40 OomScoreAdj:0 Pid:0xc8227db5e0 ReadonlyRootfs:false RemappedRoot:0xc8227db600 SeccompProfile: Sysctls:map[] UIDMapping:[] UTS:0xc8227db5f0}"
Jul 07 17:34:05 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:05.260225140Z" level=warning msg="exit status 2"
Jul 07 17:34:06 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:06.783899986Z" level=info msg="{Action=remove, ID=b807fe0c6ad2de3fe1d97c8da80def1ebdbd4960290f1963a80e018d7daeb464, Username=vagrant, LoginUID=1000, PID=14929}"
Jul 07 17:34:06 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:06.837120807Z" level=error msg="error locating sandbox id 09cb500ec04ef796a07e9bf9dfe905935ad7d0bd4525e12b1f096288275da991: sandbox 09cb500ec04ef796a07e9bf9dfe905935ad7d0bd4525e12b1f096288275da991 not found"
Jul 07 17:34:06 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:06.837958453Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/shm: invalid argument"
Jul 07 17:34:06 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:06.837980533Z" level=error msg="Error unmounting container b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b: not mounted"
Jul 07 17:34:06 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:06.839708089Z" level=error msg="Handler for POST /v1.21/containers/b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b/start returned error: Cannot start container b953ede06307b24233f110d1ab8e400c5ea1005602757d3332e0d5fc9b02db6b: [9] System error: could not synchronise with container process"
Jul 07 17:34:09 localhost.localdomain forward-journal[792]: time="2016-07-07T17:34:09.305267766Z" level=error msg="Handler for GET /containers/c880199ffdf5dead64bf1b89a8dfc440a9efe47be27c9260aa401671de352c85/json returned error: No such container: c880199ffdf5dead64bf1b89a8dfc440a9efe47be27c9260aa401671de352c85"

@mrunalp

[rhatdan]

@rhvgoyal I think this is devmapper related?

[runcom]

Weird, "System error" is usually related to libcontainer

[mrunalp]

Yeah there was one known issue with Json marshaling but we have a patch for that already. Will need to add some additional debugs for this one.

Sent from my iPhone

On Jul 7, 2016, at 2:10 PM, Antonio Murdaca [email protected] wrote:

Weird, "System error" is usually related to libcontainer

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[dustymabe]

@smarterclayton - can we close this?