hazelcast-jet icon indicating copy to clipboard operation
hazelcast-jet copied to clipboard

Published 20 hours ago • verified publisher icon hazelcast

Do proper cleanup in Docker-based tests

Open kwart opened this issue 3 years ago • 4 comments

Do proper resource cleanup in Docker-based tests.

Docker volumes and exited containers are not removed after the tests and they consume system resources.

$ docker ps -a
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS                     PORTS                                               NAMES
0efa39e27b27        debezium/example-mysql:1.2              "docker-entrypoint.s…"   2 hours ago         Exited (0) 2 hours ago                                                         affectionate_ritchie
a3fefde5c41b        debezium/example-mysql:1.2              "docker-entrypoint.s…"   2 hours ago         Exited (0) 2 hours ago                                                         nifty_golick
0f6c9e9ea4ca        debezium/example-mysql:1.2              "docker-entrypoint.s…"   2 hours ago         Exited (0) 2 hours ago                                                         sleepy_franklin
1246b3187cd8        debezium/example-mysql:1.2              "docker-entrypoint.s…"   2 hours ago         Exited (137) 2 hours ago                                                       hopeful_jackson
259841aef88c        debezium/example-mysql:1.2              "docker-entrypoint.s…"   2 hours ago         Exited (0) 2 hours ago                                                         clever_varahamihira
c6b25248a8b4        debezium/example-mysql:1.2              "docker-entrypoint.s…"   2 hours ago         Exited (0) 2 hours ago                                                         epic_spence
e801b2344e00        debezium/example-mysql:1.2              "docker-entrypoint.s…"   3 hours ago         Exited (0) 3 hours ago                                                         determined_mccarthy
f011ebea7db1        debezium/example-mysql:1.2              "docker-entrypoint.s…"   3 hours ago         Exited (0) 3 hours ago                                                         amazing_newton
2a49a2b85b38        debezium/example-mysql:1.2              "docker-entrypoint.s…"   3 hours ago         Exited (0) 3 hours ago                                                         romantic_torvalds
1d7dc98e4dc5        debezium/example-mysql:1.2              "docker-entrypoint.s…"   3 hours ago         Exited (137) 3 hours ago                                                       elegant_lumiere
a75794a5bf8e        debezium/example-mysql:1.2              "docker-entrypoint.s…"   3 hours ago         Exited (0) 3 hours ago                                                         gracious_johnson

kwart avatar Oct 08 '20 08:10 kwart

Should this have been fixed with #2590?

mtopolnik avatar Oct 08 '20 12:10 mtopolnik

I've submitted a PR (#2597) which should shed some light on the situation. It should answer two questions:

  1. what tests are instantiating the containers that don't get cleaned up?
  2. are these containers being started by code that has the previous fix (#2590) or not?

Let's see if this helps identifying the problem.

One other investigation track I have is that the listing from the issue description shows some "137" container statuses, which mean out of memory in Docker, which might explain some containers getting stuck, but I'm kind of unable to confirm anything on that path...

jbartok avatar Oct 12 '20 10:10 jbartok

I will be actively monitoring the Jenkins machine and doing manual clean-up, until I can figure out the problem.

jbartok avatar Oct 12 '20 10:10 jbartok

Not seeing stuck containers since two days now. Maybe the leftovers were from runs which didn't have the fixed code yet... Will keep monitoring it.

jbartok avatar Oct 14 '20 08:10 jbartok