balena-raspberrypi icon indicating copy to clipboard operation
balena-raspberrypi copied to clipboard

Published 20 hours ago • verified publisher icon balena-os

balenad crashed on Pi3 with "runtime/cgo: pthread_create failed: Resource temporarily unavailable"

Open saintaardvark opened this issue 5 years ago • 2 comments
trafficstars

A user reported a crash of balenad on a Pi3 with this error:

runtime/cgo: pthread_create failed: Resource temporarily unavailable

They supplied a backtrace, available on their forum post and also in JF. The first bit looks like this:

Oct 23 12:22:22 11ac67e balenad[851]: runtime/cgo: pthread_create failed: Resource temporarily unavailable
Oct 23 12:22:22 11ac67e balenad[851]: SIGABRT: abort
Oct 23 12:22:22 11ac67e balenad[851]: PC=0x76d87c46 m=19 sigcode=4294967290
Oct 23 12:22:22 11ac67e balenad[851]: goroutine 0 [idle]:
Oct 23 12:22:22 11ac67e balenad[851]: runtime: unknown pc 0x76d87c46
Oct 23 12:22:22 11ac67e balenad[851]: stack: frame={sp:0x4625d988, fp:0x0} stack=[0x45a5e1bc,0x4625ddbc)
Oct 23 12:22:22 11ac67e balenad[851]: 4625d908:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d918:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d928:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d938:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d948:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d958:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d968:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d978:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d988: <74702ed0  76d94f07  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d998:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9a8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9b8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9c8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9d8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9e8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9f8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: runtime: unknown pc 0x76d87c46
Oct 23 12:22:22 11ac67e balenad[851]: stack: frame={sp:0x4625d988, fp:0x0} stack=[0x45a5e1bc,0x4625ddbc)
Oct 23 12:22:22 11ac67e balenad[851]: 4625d908:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d918:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d928:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d938:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d948:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d958:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d968:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d978:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d988: <74702ed0  76d94f07  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d998:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9a8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9b8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9c8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9d8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9e8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: 4625d9f8:  00000000  00000000  00000000  00000000
Oct 23 12:22:22 11ac67e balenad[851]: goroutine 42 [running]:
Oct 23 12:22:22 11ac67e balenad[851]: runtime.systemstack_switch()
Oct 23 12:22:22 11ac67e balenad[851]: 	/usr/lib/go/src/runtime/asm_arm.s:294 +0x4 fp=0x130495dc sp=0x130495d8 pc=0x6c75c
Oct 23 12:22:22 11ac67e balenad[851]: runtime.gcMarkTermination(0xe7830138, 0x3fef4ad5)
Oct 23 12:22:22 11ac67e balenad[851]: 	/usr/lib/go/src/runtime/mgc.go:1647 +0x62c fp=0x13049770 sp=0x130495dc pc=0x2d13c
Oct 23 12:22:22 11ac67e balenad[851]: runtime.gcMarkDone()
Oct 23 12:22:22 11ac67e balenad[851]: 	/usr/lib/go/src/runtime/mgc.go:1513 +0x204 fp=0x13049790 sp=0x13049770 pc=0x2ca78
Oct 23 12:22:22 11ac67e balenad[851]: runtime.gcBgMarkWorker(0x13042000)
Oct 23 12:22:22 11ac67e balenad[851]: 	/usr/lib/go/src/runtime/mgc.go:1912 +0x270 fp=0x130497e4 sp=0x13049790 pc=0x2dd34
Oct 23 12:22:22 11ac67e balenad[851]: runtime.goexit()
Oct 23 12:22:22 11ac67e balenad[851]: 	/usr/lib/go/src/runtime/asm_arm.s:1015 +0x4 fp=0x130497e4 sp=0x130497e4 pc=0x6e74c
Oct 23 12:22:22 11ac67e balenad[851]: created by runtime.gcBgMarkStartWorkers
Oct 23 12:22:22 11ac67e balenad[851]: 	/usr/lib/go/src/runtime/mgc.go:1723 +0x68

I'm filing this issue here, rather than in balenad, on the assumption that this may be a problem with process limits.

From the user's description at https://forums.balena.io/t/203121:

My application uses a relatively large number of threads, so judging from the error message, I’m assuming I’m hitting a limit there. The main application I am running, is a Java (1.6u65, ancient I know) application, with a mix of threads spawning from opening serial ports and threads spawned from varying (Scheduled)ExecutorServices. Most threads are started during initialization of the application, with others being spawned to handle messages from/to the different peripheral ports, timed events and handling socket connections. Synchronization is done using block-level locks at the finest level, with different locks for different parts. It’s a bit tough going into too much detail on the forum as it’s a commercial product.

saintaardvark avatar Oct 28 '20 20:10 saintaardvark

[saintaardvark] This issue has attached support thread https://jel.ly.fish/b23a8cc0-c4fb-45b0-924c-15b8483cad7e

jellyfish-bot avatar Oct 28 '20 20:10 jellyfish-bot

I am seeing a very similar trace boil up on a raspi0w2:

Jan 12 22:02:54 9cb976c balenad[5284]: runtime/cgo: pthread_create failed: Resource temporarily unavailable
Jan 12 22:02:54 9cb976c balenad[5284]: SIGABRT: abort
Jan 12 22:02:54 9cb976c balenad[5284]: PC=0x7fb99137e8 m=0 sigcode=18446744073709551610
Jan 12 22:02:54 9cb976c balenad[5284]: goroutine 0 [idle]:
Jan 12 22:02:54 9cb976c balenad[5284]: runtime: unknown pc 0x7fb99137e8
Jan 12 22:02:54 9cb976c balenad[5284]: stack: frame={sp:0x7fe80bf1c0, fp:0x0} stack=[0x7fe78c0878,0x7fe80bf890)
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0c0:  0000007fb98e2694  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0d0:  0000007fb9b6ba30  0000007fe80bf1c8
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0e0:  0000007fe80bf1c4  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0f0:  0000007fe80bf1f0  0000007fb9b4bc94
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf100:  0000007fe80bf310  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf110:  0000007fb9b691a0  0000000000000005
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf120:  0000007fb9b741c0  0000000000000001
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf130:  0000000000000000  000000000f11ed7d
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf140:  0000000000401b70  0000007fb9b74558
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf150:  0000007fb9b72000  0000007fe80bf1c4
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf160:  0000007fe80bf1c8  00000000003c47b5
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf170:  0000000000400c30  0000007fe80bf2a8
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf180:  0000007f00000005  0000007fe80bf298
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf190:  0000007fb9b691a0  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1a0:  0000007fb9b69128  0000000000000003
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1b0:  0000000000000008  0000007fb9b4b48c
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1c0: <0000007fb9a46000  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1d0:  0000000000000000  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1e0:  0000007f00000001  0000007fb9b741c0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1f0:  0000007fe80bf2e0  0000007fb9b4fad8
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf200:  0000000002c4e1a0  0000007fb9b69dd0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf210:  0000007fb9b6a5d0  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf220:  0000000000000000  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf230:  0000000000000000  00000000019eaed0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf240:  0000000002c4e3d0  0000000002ce07a0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf250:  fffffffe7fffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf260:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf270:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf280:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf290:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf2a0:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf2b0:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: runtime: unknown pc 0x7fb99137e8
Jan 12 22:02:54 9cb976c balenad[5284]: stack: frame={sp:0x7fe80bf1c0, fp:0x0} stack=[0x7fe78c0878,0x7fe80bf890)
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0c0:  0000007fb98e2694  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0d0:  0000007fb9b6ba30  0000007fe80bf1c8
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0e0:  0000007fe80bf1c4  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf0f0:  0000007fe80bf1f0  0000007fb9b4bc94
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf100:  0000007fe80bf310  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf110:  0000007fb9b691a0  0000000000000005
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf120:  0000007fb9b741c0  0000000000000001
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf130:  0000000000000000  000000000f11ed7d
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf140:  0000000000401b70  0000007fb9b74558
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf150:  0000007fb9b72000  0000007fe80bf1c4
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf160:  0000007fe80bf1c8  00000000003c47b5
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf170:  0000000000400c30  0000007fe80bf2a8
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf180:  0000007f00000005  0000007fe80bf298
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf190:  0000007fb9b691a0  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1a0:  0000007fb9b69128  0000000000000003
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1b0:  0000000000000008  0000007fb9b4b48c
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1c0: <0000007fb9a46000  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1d0:  0000000000000000  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1e0:  0000007f00000001  0000007fb9b741c0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf1f0:  0000007fe80bf2e0  0000007fb9b4fad8
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf200:  0000000002c4e1a0  0000007fb9b69dd0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf210:  0000007fb9b6a5d0  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf220:  0000000000000000  0000000000000000
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf230:  0000000000000000  00000000019eaed0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf240:  0000000002c4e3d0  0000000002ce07a0
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf250:  fffffffe7fffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf260:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf270:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf280:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf290:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf2a0:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: 0000007fe80bf2b0:  ffffffffffffffff  ffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: goroutine 1 [running, locked to thread]:
Jan 12 22:02:54 9cb976c balenad[5284]: runtime.systemstack_switch()
Jan 12 22:02:54 9cb976c balenad[5284]:         /usr/lib/go/src/runtime/asm_arm64.s:182 +0x8 fp=0x400005e700 sp=0x400005e6f0 pc=0x45e2a0
Jan 12 22:02:54 9cb976c balenad[5284]: runtime.newproc(0x0, 0x19ead30)
Jan 12 22:02:54 9cb976c balenad[5284]:         /usr/lib/go/src/runtime/proc.go:3243 +0x58 fp=0x400005e750 sp=0x400005e700 pc=0x43ad50
Jan 12 22:02:54 9cb976c balenad[5284]: runtime.init.4()
Jan 12 22:02:54 9cb976c balenad[5284]:         /usr/lib/go/src/runtime/proc.go:239 +0x30 fp=0x400005e770 sp=0x400005e750 pc=0x432c18
Jan 12 22:02:54 9cb976c balenad[5284]: runtime.init()
Jan 12 22:02:54 9cb976c balenad[5284]:         <autogenerated>:1 +0x74 fp=0x400005e780 sp=0x400005e770 pc=0x45df9c
Jan 12 22:02:54 9cb976c balenad[5284]: runtime.main()
Jan 12 22:02:54 9cb976c balenad[5284]:         /usr/lib/go/src/runtime/proc.go:147 +0xb0 fp=0x400005e7d0 sp=0x400005e780 pc=0x4328a8
Jan 12 22:02:54 9cb976c balenad[5284]: runtime.goexit()
Jan 12 22:02:54 9cb976c balenad[5284]:         /usr/lib/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x400005e7d0 sp=0x400005e7d0 pc=0x460bcc
Jan 12 22:02:54 9cb976c balenad[5284]: r0      0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r1      0x7fe80bf1d0
Jan 12 22:02:54 9cb976c balenad[5284]: r2      0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r3      0x8
Jan 12 22:02:54 9cb976c balenad[5284]: r4      0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r5      0x6
Jan 12 22:02:54 9cb976c balenad[5284]: r6      0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r7      0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r8      0x87
Jan 12 22:02:54 9cb976c balenad[5284]: r9      0x7fe80bf1d0
Jan 12 22:02:54 9cb976c balenad[5284]: r10     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r11     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r12     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r13     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r14     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r15     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r16     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r17     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r18     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r19     0xffffffffffffffff
Jan 12 22:02:54 9cb976c balenad[5284]: r20     0x7fb9b69ed0
Jan 12 22:02:54 9cb976c balenad[5284]: r21     0x30df4450
Jan 12 22:02:54 9cb976c balenad[5284]: r22     0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r23     0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r24     0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r25     0x0
Jan 12 22:02:54 9cb976c balenad[5284]: r26     0x19eaed0
Jan 12 22:02:54 9cb976c balenad[5284]: r27     0x2c4e3d0
Jan 12 22:02:54 9cb976c balenad[5284]: r28     0x2ce07a0
Jan 12 22:02:54 9cb976c balenad[5284]: r29     0x7fe80bf2d0
Jan 12 22:02:54 9cb976c balenad[5284]: lr      0x7fb9900dd4
Jan 12 22:02:54 9cb976c balenad[5284]: sp      0x7fe80bf1c0
Jan 12 22:02:54 9cb976c balenad[5284]: pc      0x7fb99137e8
Jan 12 22:02:54 9cb976c balenad[5284]: fault   0x0
Jan 12 22:02:54 9cb976c balenad[3867]: time="2022-01-12T22:02:54.335876013Z" level=warning msg="Failed to disable IPv6 on all interfaces on network namespace \"/var/run/balena-engine/netns/487a779fa80c\": reexec to set IPv6 failed: exit >
Jan 12 22:02:56 9cb976c balenad[3885]: time="2022-01-12T22:02:56.221471150Z" level=info msg="shim reaped" id=c7a88c82c1ecd5a1b49d69ba9ced32ef622e5aefe663c11e70ac578ad7c73cd7
Jan 12 22:02:56 9cb976c balenad[3867]: time="2022-01-12T22:02:56.230170974Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 12 22:05:11 9cb976c 8b7136e5c199[3867]: [api]     GET /v1/healthy 200 - 30.603 ms
Jan 12 22:05:11 9cb976c balena-supervisor[4911]: [api]     GET /v1/healthy 200 - 30.603 ms
Jan 12 22:05:36 9cb976c 8b7136e5c199[3867]: [event]   Event: Update notification {}
Jan 12 22:05:36 9cb976c balena-supervisor[4911]: [event]   Event: Update notification {}
Jan 12 22:05:36 9cb976c 8b7136e5c199[3867]: [api]     POST /v1/update 204 - 44.340 ms
Jan 12 22:05:36 9cb976c balena-supervisor[4911]: [api]     POST /v1/update 204 - 44.340 ms
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [info]    Applying target state
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [info]    Applying target state
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dca2a4f>
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dc>
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dca2a4f>
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dc>
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [event]   Event: Delta image download {"image":{"name":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dca2a4f81@sha256:4ebbdfe28d52c9caab44eec3f7f0564ccf09dd5545bde8f676b0bfe33a182799",">
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [event]   Event: Delta image download {"image":{"name":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dca2a4f81@sha256:4ebbdfe28d52c9caab44eec3f7f0564ccf09dd5545bde8f676b0bfe33a1827>
Jan 12 22:05:37 9cb976c 8b7136e5c199[3867]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Starting delta to registry2.balen>
Jan 12 22:05:37 9cb976c balena-supervisor[4911]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Starting delta to registry2.>
Jan 12 22:05:39 9cb976c 8b7136e5c199[3867]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Applying balena delta: registry2.>
Jan 12 22:05:39 9cb976c 8b7136e5c199[3867]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Using registry auth token
Jan 12 22:05:39 9cb976c balena-supervisor[4911]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Applying balena delta: regis>
Jan 12 22:05:39 9cb976c balena-supervisor[4911]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Using registry auth token
Jan 12 22:05:49 9cb976c 8b7136e5c199[3867]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Delta applied successfully
Jan 12 22:05:49 9cb976c balena-supervisor[4911]: [debug]   delta([main] registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c): Delta applied successfully
Jan 12 22:05:49 9cb976c 8b7136e5c199[3867]: [event]   Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dca2a4f81@sha256:4ebbdfe28d52c9caab44eec3f7f0564ccf09dd5545bde8f676b0bfe33a182799","appI>
Jan 12 22:05:49 9cb976c balena-supervisor[4911]: [event]   Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/79c26b15ea60f86c1f1e265dca2a4f81@sha256:4ebbdfe28d52c9caab44eec3f7f0564ccf09dd5545bde8f676b0bfe33a182799",>
Jan 12 22:05:50 9cb976c 8b7136e5c199[3867]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:50 9cb976c balena-supervisor[4911]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:50 9cb976c 8b7136e5c199[3867]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:50 9cb976c balena-supervisor[4911]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:50 9cb976c 8b7136e5c199[3867]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:50 9cb976c balena-supervisor[4911]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:50 9cb976c 8b7136e5c199[3867]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:50 9cb976c balena-supervisor[4911]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:50 9cb976c 8b7136e5c199[3867]: [event]   Event: Service kill {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"147abfc7dd58ba455d0ccc2177d411d2","releaseId":2036321}}
Jan 12 22:05:50 9cb976c balena-supervisor[4911]: [event]   Event: Service kill {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"147abfc7dd58ba455d0ccc2177d411d2","releaseId":2036321}}
Jan 12 22:05:51 9cb976c 8b7136e5c199[3867]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:51 9cb976c balena-supervisor[4911]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:51 9cb976c balena-supervisor[4911]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:51 9cb976c 8b7136e5c199[3867]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:52 9cb976c systemd[1]: docker-347087061fa42db99234289aa4360909eed4f4bb871d03bf77ea24975121f8cc.scope: Consumed 8.081s CPU time.
Jan 12 22:05:53 9cb976c balenad[3885]: time="2022-01-12T22:05:53.570400750Z" level=info msg="shim reaped" id=347087061fa42db99234289aa4360909eed4f4bb871d03bf77ea24975121f8cc
Jan 12 22:05:53 9cb976c balenad[3867]: time="2022-01-12T22:05:53.578568547Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 12 22:05:53 9cb976c balenad[3867]: time="2022-01-12T22:05:53.584167199Z" level=warning msg="error locating sandbox id 248e3f0355c3f564078153240a0b501b23fc1cc09566393d10f7609012de4027: sandbox 248e3f0355c3f564078153240a0b501b23fc1cc09>
Jan 12 22:05:53 9cb976c 8b7136e5c199[3867]: [event]   Event: Service exit {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"147abfc7dd58ba455d0ccc2177d411d2","releaseId":2036321}}
Jan 12 22:05:53 9cb976c balena-supervisor[4911]: [event]   Event: Service exit {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"147abfc7dd58ba455d0ccc2177d411d2","releaseId":2036321}}
Jan 12 22:05:53 9cb976c 8b7136e5c199[3867]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:53 9cb976c balena-supervisor[4911]: [debug]   Replacing container for service main because of config changes:
Jan 12 22:05:53 9cb976c 8b7136e5c199[3867]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:53 9cb976c balena-supervisor[4911]: [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:1e98815c375c7d20d164ed13c4a09aeb1cf7580e9f527210895e3083c63016ef"}}
Jan 12 22:05:53 9cb976c 8b7136e5c199[3867]: [event]   Event: Service stop {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"147abfc7dd58ba455d0ccc2177d411d2","releaseId":2036321}}
Jan 12 22:05:53 9cb976c balena-supervisor[4911]: [event]   Event: Service stop {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"147abfc7dd58ba455d0ccc2177d411d2","releaseId":2036321}}
Jan 12 22:05:55 9cb976c 8b7136e5c199[3867]: [event]   Event: Image removal {"image":{"name":"registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c","appId":>
Jan 12 22:05:55 9cb976c balena-supervisor[4911]: [event]   Event: Image removal {"image":{"name":"registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c","ap>
Jan 12 22:05:55 9cb976c 8b7136e5c199[3867]: [event]   Event: Image removed {"image":{"name":"registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c","appId":>
Jan 12 22:05:55 9cb976c balena-supervisor[4911]: [event]   Event: Image removed {"image":{"name":"registry2.balena-cloud.com/v2/0cbed30397cb835c0e60d6a3f6b9110a@sha256:538a29849e7baa7e3dd6dc9f2ca0d750ef4f733bd507d77ec22846bc87190b6c","ap>
Jan 12 22:05:56 9cb976c 8b7136e5c199[3867]: [event]   Event: Service install {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:05:56 9cb976c balena-supervisor[4911]: [event]   Event: Service install {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:05:57 9cb976c 8b7136e5c199[3867]: [event]   Event: Service installed {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:05:57 9cb976c balena-supervisor[4911]: [event]   Event: Service installed {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:05:57 9cb976c 8b7136e5c199[3867]: [event]   Event: Service start {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:05:57 9cb976c balena-supervisor[4911]: [event]   Event: Service start {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:05:57 9cb976c balenad[3885]: time="2022-01-12T22:05:57.530827333Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/109e7c86d4b2ad982e08a780b76c6985041348c4a49f6a648eb83d28f25308e8.sock debug>
Jan 12 22:05:57 9cb976c balenad[3885]: time="2022-01-12T22:05:57.758687961Z" level=info msg="shim balena-engine-containerd-shim started" address=/containerd-shim/e27cb6508da2f0dc78b63f875d840c2bc5963c309af18f6bcbdb5c954574ca26.sock debug>
Jan 12 22:05:59 9cb976c balenad[3885]: time="2022-01-12T22:05:59.751618571Z" level=error msg="OCI runtime state failed: balena-engine-runc did not terminate sucessfully: runtime/cgo: pthread_create failed: Resource temporarily unavailabl>
Jan 12 22:05:59 9cb976c balenad[3885]: time="2022-01-12T22:05:59.795443054Z" level=info msg="shim reaped" id=693a277b64a7b33f40706a95c79c644157d920903e5cffb1a0ee8a4d7bd76e69
Jan 12 22:05:59 9cb976c balenad[3867]: time="2022-01-12T22:05:59.879696737Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 12 22:05:59 9cb976c balenad[3867]: time="2022-01-12T22:05:59.879703248Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 12 22:06:00 9cb976c balenad[3867]: time="2022-01-12T22:06:00.139448614Z" level=error msg="693a277b64a7b33f40706a95c79c644157d920903e5cffb1a0ee8a4d7bd76e69 cleanup: failed to delete container from containerd: no such container"
Jan 12 22:06:00 9cb976c balenad[3867]: time="2022-01-12T22:06:00.250910303Z" level=error msg="Handler for POST /v1.40/containers/693a277b64a7b33f40706a95c79c644157d920903e5cffb1a0ee8a4d7bd76e69/start returned error: OCI runtime create fa>
Jan 12 22:06:00 9cb976c balenad[5732]: balena: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running >
Jan 12 22:06:00 9cb976c 8b7136e5c199[3867]: [event]   Event: Service started {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:06:00 9cb976c balena-supervisor[4911]: [event]   Event: Service started {"service":{"appId":1753619,"serviceId":727635,"serviceName":"main","commit":"c60b6c83b3da6dd154286636d38c5f36","releaseId":2037418}}
Jan 12 22:06:01 9cb976c balena-supervisor[4911]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2022-01-12 22:06:01 -o json CONTAINER_ID_FULL=34861af81972955f1a2c4c638d637b69c9c997d80d20c153faf68cd85f1c5ffa
Jan 12 22:06:01 9cb976c 8b7136e5c199[3867]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2022-01-12 22:06:01 -o json CONTAINER_ID_FULL=34861af81972955f1a2c4c638d637b69c9c997d80d20c153faf68cd85f1c5ffa
Jan 12 22:06:01 9cb976c systemd-udevd[28]: starting version 232
Jan 12 22:06:02 9cb976c systemd-udevd[5936]: Using default interface naming scheme 'v243'.
Jan 12 22:06:03 9cb976c systemd-udevd[5939]: Using default interface naming scheme 'v243'.
Jan 12 22:06:03 9cb976c systemd-udevd[5927]: Using default interface naming scheme 'v243'.
Jan 12 22:06:03 9cb976c systemd-udevd[5917]: Using default interface naming scheme 'v243'.
Jan 12 22:06:03 9cb976c systemd-udevd[5939]: ptmx: Failed to process device, ignoring: File exists
Jan 12 22:06:04 9cb976c balena-supervisor[4911]: [debug]   Finished applying target state
Jan 12 22:06:04 9cb976c 8b7136e5c199[3867]: [debug]   Finished applying target state
Jan 12 22:06:04 9cb976c 8b7136e5c199[3867]: [success] Device state apply success
Jan 12 22:06:04 9cb976c balena-supervisor[4911]: [success] Device state apply success

It results in a SIGTERM getting sent to the application. Any leads to chase here?

This is on a fresh install with the most recent versions...

meawoppl avatar Jan 12 '22 22:01 meawoppl