bugs icon indicating copy to clipboard operation
bugs copied to clipboard

Published 20 hours ago • verified publisher icon coreos

Unable to start container using low memory limit

Open chrischdi opened this issue 6 years ago • 3 comments

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.6.0
VERSION_ID=1967.6.0
BUILD_ID=2019-02-12-2138
PRETTY_NAME="Container Linux by CoreOS 1967.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr

Environment

We're running in Openstack / VMWare Integrated Openstack:

$ cat /proc/meminfo  | grep MemTotal
MemTotal:        8171244 kB

Expected Behavior

Docker container starts

Actual Behavior

Docker fails to start a container

Reproduction Steps

  1. Start docker container including memory limit: 
    docker run --rm --memory 10485760 -ti jimmidyson/configmap-reload:v0.2.2

/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "process_linux.go:367: setting cgroup config for procHooks process caused \"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/aa95f28a0ecfc178c4fb52e361281f936f2e556679f55ae193aacd5b150a6416/memory.limit_in_bytes: device or resource busy\""": unknown.


When using a higher memory limit, starting the container is no problem.
When using a slightly higher memory limit, the result is varying:

$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2 /run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "process_linux.go:367: setting cgroup config for procHooks process caused \"failed to write 12534336 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/5ea6580804d337da46c06f4c06b4df7dce62adb17362687ac3d0faabe966f302/memory.limit_in_bytes: device or resource busy\""": unknown.
$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2 /run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: docker-runc did not terminate sucessfully: panic: No error following JSON procError payload. [recovered]
panic: No error following JSON procError payload.

goroutine 1 [running]: github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc420145870) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23d
panic(0x55f49ba9ce40, 0x55f49bb1d5c0) /usr/lib/go1.10/src/runtime/panic.go:502 +0x22d github.com/opencontainers/runc/libcontainer.parseSync(0x55f49bb1f500, 0xc4200da108, 0xc420144a50, 0x3, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2f3
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc4200e0700, 0x0, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/process_linux.go:323 +0x3b4
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc4201ae000, 0xc4200e6d80, 0x1, 0x0, 0xc420144c50) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/container_linux.go:343 +0x95
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc4201ae000, 0xc4200e6d80, 0x0, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/container_linux.go:237 +0xc5
main.(*runner).run(0xc420145398, 0xc4200e49c0, 0x0, 0x0, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/utils_linux.go:303 +0x90c
main.startContainer(0xc420124140, 0xc4201260e0, 0x1, 0x0, 0x0, 0x20, 0xc4200d2cf0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/utils_linux.go:428 +0x3c4
main.glob..func2(0xc420124140, 0x0, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/create.go:65 +0xb0
reflect.Value.call(0x55f49baa85c0, 0x55f49bb1cb10, 0x13, 0x55f49b7d9e57, 0x4, 0xc420055830, 0x1, 0x1, 0xc4200600c0, 0x55f49bb10e20, ...) /usr/lib/go1.10/src/reflect/value.go:447 +0x96b reflect.Value.Call(0x55f49baa85c0, 0x55f49bb1cb10, 0x13, 0xc420055830, 0x1, 0x1, 0x55f49b79d4b8, 0x55f49b7d9f77, 0x4) /usr/lib/go1.10/src/reflect/value.go:308 +0xa6 github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x55f49baa85c0, 0x55f49bb1cb10, 0xc420124140, 0x0, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x16f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x55f49b7da66b, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55f49b7de64e, 0x12, 0x0, ...)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x9ac
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc420102180, 0xc4200100f0, 0xf, 0xf, 0x0, 0x0) /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x548
main.main() /build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/main.go:147 +0xaea
: unknown. $ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2 2019/02/18 07:16:16 Missing volume-dir 2019/02/18 07:16:16 Usage of /configmap-reload: -volume-dir value the config map volume directory to watch for updates; may be used multiple times -webhook-method string the HTTP method url to use to send the webhook (default "POST") -webhook-status-code int the HTTP status code indicating successful triggering of reload (default 200) -webhook-url string the url to send a request to when the specified config map volume directory has been updated


### Other Information ###

Unable to reproduce the problem on same os configuration but using the following CoreOS Release:

$ cat /etc/os-release NAME="Container Linux by CoreOS" ID=coreos VERSION=1967.3.0 VERSION_ID=1967.3.0 BUILD_ID=2019-01-08-0044 PRETTY_NAME="Container Linux by CoreOS 1967.3.0 (Rhyolite)" ANSI_COLOR="38;5;75" HOME_URL="https://coreos.com/" BUG_REPORT_URL="https://issues.coreos.com" COREOS_BOARD="amd64-usr"

We did not test `1967.5.0`

chrischdi avatar Feb 18 '19 07:02 chrischdi

Also tested 1967.5.0: so it looks to me that the runc fix is producing this bug. Because I can also reproduce the problem on 1967.5.0

One more note: we are using docker 18.06.1-ce, installed via torcx.

chrischdi avatar Feb 18 '19 08:02 chrischdi

You would probably want to look into the upstream issue, which is still in discussions: https://github.com/opencontainers/runc/issues/1980

dongsupark avatar Feb 18 '19 10:02 dongsupark

Hi again,

the problem should be resolved by https://github.com/opencontainers/runc/pull/1984 and https://github.com/opencontainers/runc/pull/2006 . Will these patches also get into a new CoreOS Release?

chrischdi avatar Mar 12 '19 12:03 chrischdi