balena-engine
balena-engine copied to clipboard
Published
20 hours ago •
balena-os
balena-os
balena-engine warnings while starting up
Description
There are a couple of warnings that appear when balena-engine starts up.
Removing stale sandbox and could not create bridge network for id
Feb 06 15:13:13 8e1b3fb balenad[762]: time="2019-02-06T15:13:13.660459008Z" level=warning msg="could not create bridge network for id 9df83c1fb24101b00e6329fb3f291a4d14dd40ec1e09d459f6b479dfac553bc8 bridge name docker0 while booting up from persistent state: bridge device with non default name docker0 must be created manually"
Feb 06 15:13:14 8e1b3fb balenad[762]: time="2019-02-06T15:13:14.082873383Z" level=info msg="Removing stale sandbox fbe700c676b33178b5ca105af1f69b82aa8b64161482ac97c3d6be24e49d81f8 (3aed15a3fb126f9f53aef1d80970d2c96d3a7a0132b77d52fe9ee494f14255ab)"
Feb 06 15:13:14 8e1b3fb balenad[762]: time="2019-02-06T15:13:14.144530049Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 9516f16d2c928f10bf063b040f39799c5697324edb5cba89c5b01584661a8d11 9add72e8385f957377bdebef30ac68660ae51109ad9d2c03999d96b2be36c62d], retrying...."
Feb 06 15:13:15 8e1b3fb balenad[762]: time="2019-02-06T15:13:15.598632965Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
These delay balena-engine from starting up for a few seconds on a pi0.
Steps to reproduce the issue:
- Flash balenaOS 2.29.2 production on a pi3.
- Push a container
- Reboot and check
journalctl -u balena
Describe the results you received: Some warnings as above delay balena-engine from starting
Describe the results you expected: Not a few seconds of delay on a pi0
Additional information you deem important (e.g. issue happens only occasionally):
Output of balena-engine version:
balena-engine version 17.12.0-dev, build dceb2fc48071b78a8a828e0468a15a479515385f
Output of balena-engine info:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 2
Server Version: 17.12.0-dev
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 38
Dirperm1 Supported: true
Logging Driver: journald
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan null
Log: journald json-file
Swarm:
NodeID:
Is Manager: false
Node Address:
Runtimes: bare runc
Default Runtime: runc
Init Binary: balena-engine-init
containerd version:
runc version: 13e66eedaddfbfeda2a73d23701000e4e63b5471
init version: N/A (expected: )
Kernel Version: 4.14.79
Operating System: balenaOS 2.29.2+rev2
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 975.7MiB
Name: 8e1b3fb
ID: OOA2:JLHW:J7ZM:FTCI:JODP:OF6L:PRPS:ST2L:DXPK:2WU4:7KKX:EAE6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
@robertgzr petros asked me to add a debug log snippet here https://pastebin.com/MmiU2gXG
This is a pi0w, 2.29.2, debug : true via daemon.json and then a systemctl restart balena
@ZubairLK I narrowed down the code producing the "Removing stale sandbox" logs to https://github.com/balena-os/balena-engine/blob/17.12-resin/daemon/container_operations.go#L504 and https://github.com/balena-os/balena-libnetwork/blob/17.12-resin/sandbox_store.go#L279
It seems to be a default operation the daemon does when starting a container as part of setting up the network, basically cleaning after a previous run of the container. I'm not sure why this would be necessary when we start from a fresh install though.
The warning about not being able to find the docker0 bridge also seems related to starting up previously existing containers. The networkId gets stored as part of the container metadata... which makes it look like containers were previously started under docker and now balenaEngine brings them up the network configuration tells it to look for the configured bridge (docker0 instead of balena0) :thinking:
So at @robertgzr what is the plan on this one? Do we have an idea on a way forward?
@shaunmulligan @ZubairLK is it possible to capture the state of /var/lib/balena before balenaEngine prints these warnings, ideally from the first boot (before it had any chance to start containers/set up networks). Just to see if there's anything interesting there...
Another angle would be to look at the library used by the supervisor to talk to the docker API (since we're not calling out to the balena-engine cli, right?)
I have never seen these logs on a non-balenaOS device, which makes me think it's related to other OS components creating container configurations which are not valid.
@robertgzr you might be on to something. Removing the docker folder and restarting balena-engine doesn't show those warnings.
root@2801ec1:~# rm /var/lib/docker/* -r
but when the supervisor container is started and it downloads the container, you do see the warnings except for the docker0 line. i.e. still see
Feb 06 15:13:14 8e1b3fb balenad[762]: time="2019-02-06T15:13:14.144530049Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 9516f16d2c928f10bf063b040f39799c5697324edb5cba89c5b01584661a8d11 9add72e8385f957377bdebef30ac68660ae51109ad9d2c03999d96b2be36c62d], retrying...."
I'm going to try to narrow down if any container makes them appear or specifically the supervisor
@robertgzr so the docker0 bridge warnings came via balenaOS. https://github.com/balena-os/meta-balena/pull/1449 removes them.
Now for the stale sandbox ones. They are worse in a multi-container setup. I'll see if manually creating these containers causes them as well. compared to the supervisor creating these containers.
The other stale sandbox ones are still there.
Mar 26 12:53:03 2801ec1 balenad[4689]: time="2019-03-26T12:53:03.758214852Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 60567beb11a5aad3b2c2e5eeea1c5a464e1397a63d744f0fcae393210c2d34de c5576226f63ff543ec749a1017bd706d836a8de29d3e2dd879aca37b8c428461], retrying...."
Mar 26 12:53:03 2801ec1 balenad[4689]: time="2019-03-26T12:53:03.688356116Z" level=info msg="Removing stale sandbox e132eb3a5f2cc9bd48c1111c22420bd97657948599ff4053dcc16ecb0621985c (b0951dd965fed4c0cda600deea932decbe312c884fe7287cd5c4fc08294edf2f)"
Mar 26 12:53:03 2801ec1 balenad[4689]: time="2019-03-26T12:53:03.242938138Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 60567beb11a5aad3b2c2e5eeea1c5a464e1397a63d744f0fcae393210c2d34de ccccb24f91becb883ceb0922504630e0a928680037eb7148c949bc2f3af6eddf], retrying...."
Mar 26 12:53:03 2801ec1 balenad[4689]: time="2019-03-26T12:53:03.223985953Z" level=info msg="Removing stale sandbox 58c35eec17197e099f0fd8a69cf3359b76e85d0bc4616b8828cf0c5e45026acb (12b2f1cab66e69c9a553cea028c7eb5d7117f3f0ddad1d1240aeeeb5163de6ec)"
Mar 26 12:53:02 2801ec1 balenad[4689]: time="2019-03-26T12:53:02.796633917Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 60567beb11a5aad3b2c2e5eeea1c5a464e1397a63d744f0fcae393210c2d34de 5d52fe96cc1c7eb047da256625bf460aab4a1f511f3c2b9ef8f50d25b57ebd5a], retrying...."
Mar 26 12:53:02 2801ec1 balenad[4689]: time="2019-03-26T12:53:02.778485736Z" level=info msg="Removing stale sandbox 1617122210fde2cf107366779a386921d8dffbc12432ac508727ab8891ec53c4 (dc3278c00840167a57f40792ba535cd673ce9fd19d9169e144a00790f19bdb6e)"
Mar 26 12:53:02 2801ec1 balenad[4689]: time="2019-03-26T12:53:02.351224637Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 9f207e4be460756e0ff4b5f9bd9b94ace61ca43c14d52511df3d7ed9bb24c589 b5c0f0b147213846bb36179f14b5b2a44f1737831675685e7353b0e8e10b6293], retrying...."
Mar 26 12:53:02 2801ec1 balenad[4689]: time="2019-03-26T12:53:02.340159782Z" level=info msg="Removing stale sandbox 029f4384783d104ac0f481f0c2cf889ab58531dbba03c391e84833d98d05f9b8 (6144a267188952713916d21515e2989b085de1ec1cbe6ce58faf7f632e478787)"
Mar 26 12:52:59 2801ec1 balenad[4689]: time="2019-03-26T12:52:59.897550920Z" level=info msg="ignoring event" module
=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
@robertgzr ok so this warning one I think is coming from balena-engine itself.
On a pi3, running 17.3.3
root@2801ec1:/mnt/data/test# balena version
Client:
Version: 17.13.3-dev
API version: 1.35
Go version: go1.9.7
Git commit: fe78e2c9a69313007c53c83fff4b5525fbc2ba45
Built: Mon Mar 18 10:45:02 2019
OS/Arch: linux/arm
Experimental: false
Orchestrator: swarm
I stop and remove the supervisor. Then I just run a simple sleeping container. Here is the bash history.
126 systemctl stop resin-supervisor
127 systemctl stop balena
129 rm /var/lib/docker/* -r
131 systemctl start balena
134 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
136 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
137 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
138 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
139 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
140 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
141 balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
143 systemctl restart balena
145 journalctl -u balena -r > /mnt/data/balena.log
In the journalctl log when restarting balena, there are tons of warnings.
Mar 26 15:02:30 2801ec1 balenad[14331]: time="2019-03-26T15:02:30.637277697Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/create type="*events.ContainerCreate"
Mar 26 15:02:29 2801ec1 balenad[14331]: time="2019-03-26T15:02:29.825759645Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a 27de64f0b75c658fd6041548dedcb79cf0d9da0c5f5afe8617f5ad1ea8d40151], retrying...."
Mar 26 15:02:29 2801ec1 balenad[14331]: time="2019-03-26T15:02:29.805952175Z" level=info msg="Removing stale sandbox a138a68a1ffaf6968f793349158860e07bf36e51eb9ad181e9b7a2e121bd0218 (d5375dff8606b488995f5030a24896ce7ce771ee3c3a64168daba728d644435b)"
Mar 26 15:02:29 2801ec1 balenad[14331]: time="2019-03-26T15:02:29.376579022Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a 6f9c5bc9082ee0cf8cab2856ab7151fe7ecf09c0a4f90d06724e563a86c0dc0f], retrying...."
Mar 26 15:02:29 2801ec1 balenad[14331]: time="2019-03-26T15:02:29.360028397Z" level=info msg="Removing stale sandbox 43f06309d7872ad0a61bf047e6ca35b7508934b602535999872c9ae85fdabf84 (77833c01d06e200f1c4a426e444f67583a7682666d798c76ac377881ccd3b820)"
Mar 26 15:02:28 2801ec1 balenad[14331]: time="2019-03-26T15:02:28.920782524Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a 0e472a0d0b97ffff0ef9a52f563d9c024f3bb96d34c669a07017fa03a1308177], retrying...."
Mar 26 15:02:28 2801ec1 balenad[14331]: time="2019-03-26T15:02:28.903272845Z" level=info msg="Removing stale sandbox 2a469b717996b1cc23b898166d3725f22cf91dfd30edf6cfee40162141e88927 (be9aa3adc0df30d90545d10d93482e5857b59b7adbb158fc02faf8a220dd741a)"
Mar 26 15:02:28 2801ec1 balenad[14331]: time="2019-03-26T15:02:28.463035106Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a 1de2e28ba83a05b92e500193840cd83d18f626d20de7fa371e3259bdef6f9d22], retrying...."
Mar 26 15:02:28 2801ec1 balenad[14331]: time="2019-03-26T15:02:28.445766154Z" level=info msg="Removing stale sandbox 0230ed511fd24f405f28b9df80819927e73e7da36052357b1e01788b5da183c5 (1cd523263efdbc2b7edb539ed515ad69578c345b4a4649d39ed9ea465575101f)"
Mar 26 15:02:27 2801ec1 balenad[14331]: time="2019-03-26T15:02:27.986706457Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a 5b1de96e899a65eb0dcfe7e53ef6d6cbfd97ec13e31b6f121feb0b224453ff5d], retrying...."
Mar 26 15:02:27 2801ec1 balenad[14331]: time="2019-03-26T15:02:27.969130528Z" level=info msg="Removing stale sandbox fd137420ea19272bfd1da17f72bd036e5d61be4ed8f56716b1e72c77c57732fe (936645cfa132b0d65328de9ba3eb63d5b1d58bfbf3ca53c0c6bf4a7432363905)"
Mar 26 15:02:27 2801ec1 balenad[14331]: time="2019-03-26T15:02:27.488268849Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a d1c2a5153f5d39615b101f7da1fb5bd748f16432128ebe19264aede5b74efe02], retrying...."
Mar 26 15:02:27 2801ec1 balenad[14331]: time="2019-03-26T15:02:27.470264695Z" level=info msg="Removing stale sandbox f56deda761ce8c3be394bed883c7894d29ab7fd97b5404fd12fb148a405ca056 (df7952411f37a1f61b6e4adad8cf185f11fb075f228686e94eb7d6246178d8f0)"
Mar 26 15:02:26 2801ec1 balenad[14331]: time="2019-03-26T15:02:26.978428066Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 297fe258a8ae0a27866108b3ed34b4daacd1f47dee92c4bfa39608cb9799208a 9dcf7ecc2a6f278cf6b21f55b105907dab4951d467fa7834489afe84a77f4dd1], retrying...."
Mar 26 15:02:26 2801ec1 balenad[14331]: time="2019-03-26T15:02:26.949355475Z" level=info msg="Removing stale sandbox bd9ab910096b07aa7471d97b4f0252900a4ca87c2400501f797923ae7500bc54 (56024c8cd82855b0eef65cf74b07957b4f57bde7fa8946892cc19e0a8e511a10)"
Mar 26 15:02:24 2801ec1 balenad[14331]: time="2019-03-26T15:02:24.514815329Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete"
Mar 26 15:02:24 2801ec1 balenad[14331]: time="2019-03-26T15:02:24.513631747Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/containers/
Same on 18.09.3 unfortunately.
After a rm /var/lib/docker/*, they appear after multiple invocations of a simple sleeping container.
balena run -d --restart always balenalib/raspberrypi3-debian /bin/sleep infinity
root@2801ec1:/mnt/data# balena version
Client:
Version: v18.09.3-rc1-205-g35e5e4ab4
API version: 1.39
Go version: go1.10.8
Git commit: 35e5e4ab4
Built: Thu Mar 21 15:13:09 2019
OS/Arch: linux/arm
Experimental: false
Server:
Engine:
Version: v18.09.3-rc1-205-g35e5e4ab4
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 35e5e4ab4
Built: Thu Mar 21 15:13:09 2019
OS/Arch: linux/arm
Experimental: true
root@2801ec1:/mnt/data#
-- Logs begin at Wed 2019-03-13 15:29:29 UTC, end at Tue 2019-03-26 15:42:13 UTC. --
Mar 26 15:42:13 2801ec1 balenad[18156]: time="2019-03-26T15:42:13.066886803Z" level=info msg="API listen on /var/run/balena-engine.sock"
Mar 26 15:42:13 2801ec1 balenad[18156]: time="2019-03-26T15:42:13.066840136Z" level=info msg="API listen on /var/run/balena.sock"
Mar 26 15:42:13 2801ec1 systemd[1]: Started Balena Application Container Engine.
Mar 26 15:42:13 2801ec1 balenad[18156]: time="2019-03-26T15:42:13.065409317Z" level=info msg="API listen on /var/run/balena-engine.sock"
Mar 26 15:42:13 2801ec1 balenad[18156]: time="2019-03-26T15:42:13.065033018Z" level=info msg="API listen on [::]:2375"
Mar 26 15:42:12 2801ec1 balenad[18156]: time="2019-03-26T15:42:12.991526416Z" level=info msg="Daemon has completed initialization"
Mar 26 15:42:12 2801ec1 balenad[18156]: time="2019-03-26T15:42:12.991404802Z" level=info msg="Docker daemon" commit=35e5e4ab4 graphdriver(s)=aufs version=v18.09.3-rc1-205-g35e5e4ab4
Mar 26 15:42:12 2801ec1 balenad[18156]: time="2019-03-26T15:42:12.834775803Z" level=info msg="Loading containers: done."
Mar 26 15:42:09 2801ec1 balenad[18156]: time="2019-03-26T15:42:09.042858605Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/4f985df5cd0d0c1d3fd9f2a1623c2905bffc1f8461d76c315cadd2cbf7668cdb/shim.sock debug=false pid=18550
Mar 26 15:42:09 2801ec1 balenad[18156]: time="2019-03-26T15:42:09.002262267Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/351a7167077728eeb2f747c1926868ae9a64538e1fba1a312d856eac7eaa6e4f/shim.sock debug=false pid=18543
Mar 26 15:42:08 2801ec1 balenad[18156]: time="2019-03-26T15:42:08.838012610Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/39971d4bcf38994c5705ee60c483c0cc3a555a65d2e4cda4bad34bfe01de3335/shim.sock debug=false pid=18520
Mar 26 15:42:08 2801ec1 balenad[18156]: time="2019-03-26T15:42:08.831219340Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/dd19db907b572b1fb4f33b7825abefd4da77767771815a639126d40e19051b96/shim.sock debug=false pid=18518
Mar 26 15:42:08 2801ec1 balenad[18156]: time="2019-03-26T15:42:08.490936073Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/d1b041458271d85d4748bafc49af5350c683753ff597376707f10273c82512d2/shim.sock debug=false pid=18481
Mar 26 15:42:08 2801ec1 balenad[18156]: time="2019-03-26T15:42:08.445153117Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/9cf30c381b4122bb4ee6c699b17359133a82517bfb74bba946b03ebe1e338ca4/shim.sock debug=false pid=18473
Mar 26 15:42:08 2801ec1 balenad[18156]: time="2019-03-26T15:42:08.196037016Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/moby/f0113c70522e59f1083dc713b22201accdc3284a5c1a2aa68a5829a95e948bfa/shim.sock debug=false pid=18452
Mar 26 15:42:07 2801ec1 balenad[18156]: time="2019-03-26T15:42:07.205638700Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 a49d1e29f7659fe1c79ce363d1849fb0680db3ffa2756386b0be3c059ce076db], retrying...."
Mar 26 15:42:07 2801ec1 balenad[18156]: time="2019-03-26T15:42:07.191330449Z" level=info msg="Removing stale sandbox 76a97c9e764ffcf75ada6e1920598accbdb48f8653123e8a80c6ca8d9122c5f3 (4f985df5cd0d0c1d3fd9f2a1623c2905bffc1f8461d76c315cadd2cbf7668cdb)"
Mar 26 15:42:06 2801ec1 balenad[18156]: time="2019-03-26T15:42:06.548720642Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 69c8e89836c07377247c54847e1dbc3cd23a99b46503cf05162db69b69ab986b], retrying...."
Mar 26 15:42:06 2801ec1 balenad[18156]: time="2019-03-26T15:42:06.532906884Z" level=info msg="Removing stale sandbox 5b81c1f82c4ae1188a3bdd547e58a315a86bb9502ab1b57cee1d339b5a5fb7b6 (d1b041458271d85d4748bafc49af5350c683753ff597376707f10273c82512d2)"
Mar 26 15:42:05 2801ec1 balenad[18156]: time="2019-03-26T15:42:05.877802872Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 b1e3a7f7bcc6d00e50beaf7605914d452c0da85ba8e6a657769bd4623100601d], retrying...."
Mar 26 15:42:05 2801ec1 balenad[18156]: time="2019-03-26T15:42:05.862495828Z" level=info msg="Removing stale sandbox 1754e3e5857a8c07d4025dceb411a7661c1fbb588c348e1db8429a2ed4e8ea71 (dd19db907b572b1fb4f33b7825abefd4da77767771815a639126d40e19051b96)"
Mar 26 15:42:05 2801ec1 balenad[18156]: time="2019-03-26T15:42:05.261819066Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 ad310adc116b4a1a88515028968eccc32d848c8929ad908db8c5ed973144b11d], retrying...."
Mar 26 15:42:05 2801ec1 balenad[18156]: time="2019-03-26T15:42:05.244980109Z" level=info msg="Removing stale sandbox 12bee7165fa5062e776ae7c61241761f14e58bdf3c38b26fb83ff99503d1f1d6 (39971d4bcf38994c5705ee60c483c0cc3a555a65d2e4cda4bad34bfe01de3335)"
Mar 26 15:42:04 2801ec1 balenad[18156]: time="2019-03-26T15:42:04.714814717Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 2729129de7de6455b8dabbc1587645c02fc7c32ec71e4b4c97877cc2245e78aa], retrying...."
Mar 26 15:42:04 2801ec1 balenad[18156]: time="2019-03-26T15:42:04.699986002Z" level=info msg="Removing stale sandbox c6b27f1c9090c7a0b3fe8f5ba95658dd38a3a2a75beb5d1d52ceaf0ca7c44d0f (f0113c70522e59f1083dc713b22201accdc3284a5c1a2aa68a5829a95e948bfa)"
Mar 26 15:42:04 2801ec1 balenad[18156]: time="2019-03-26T15:42:04.127976523Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 fa97e3c98c0fc9c755ce60030da2d5eaa3e21d81c04d558db5b00c7c0de16fd7], retrying...."
Mar 26 15:42:04 2801ec1 balenad[18156]: time="2019-03-26T15:42:04.114043008Z" level=info msg="Removing stale sandbox 952a09da7c807c91291fac0f17242d46bbe627a207e640ee357ae9c2adc24ee6 (9cf30c381b4122bb4ee6c699b17359133a82517bfb74bba946b03ebe1e338ca4)"
Mar 26 15:42:03 2801ec1 balenad[18156]: time="2019-03-26T15:42:03.544231685Z" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 56d923e076640c468f399d8390c7a0de5232a6151bba9f7448db25dc648edaa9 af5b7c77ea520d45df02b70bfdee4429349cd30b41a8672ff7ea6917ad799ede], retrying...."
Mar 26 15:42:03 2801ec1 balenad[18156]: time="2019-03-26T15:42:03.527399968Z" level=info msg="Removing stale sandbox 87bdb60795d77450bdce0b58726e2e86046fccbfd157d2e4819a54f12183eaec (351a7167077728eeb2f747c1926868ae9a64538e1fba1a312d856eac7eaa6e4f)"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.376225563Z" level=info msg="Loading containers: start."
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.375185417Z" level=warning msg="mountpoint for pids not found"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.374630370Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.374325893Z" level=warning msg="Your kernel does not support cgroup rt period"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.372862261Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.318692665Z" level=info msg="containerd successfully booted in 0.028328s"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.318501261Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.318231107Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.317240804Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.317078149Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.316897005Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.316718830Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.316340812Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.316176074Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.315994878Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.315831755Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.315666027Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.315466758Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.315304155Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.315138949Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.314968846Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.314589735Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.314273436Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.307953964Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.307600842Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.306831631Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.306641164Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.306459967Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.306283354Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.306103408Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.305925910Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.305728985Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.305545288Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.305277687Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.305069668Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.304447382Z" level=info msg="metadata content store policy set" policy=shared
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.304259051Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.303699160Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.303351455Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.296715684Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.296433082Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.294691067Z" level=info msg="starting containerd" revision= version=1.2.2+unknown
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.164081876Z" level=info msg="libcontainerd: started new balena-engine-containerd process" pid=18176
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.150652366Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Mar 26 15:42:00 2801ec1 balenad[18156]: time="2019-03-26T15:42:00.147139170Z" level=warning msg="Running experimental build"
Mar 26 15:41:59 2801ec1 systemd[1]: Starting Balena Application Container Engine...
[xginn8] This issue has attached support thread https://jel.ly.fish/#/1106b429-94a1-46aa-969e-7a86f0075406
