yet-another-docker-plugin

yet-another-docker-plugin copied to clipboard

The containers not getting cleaned. any suggest whats going wrong?

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3002f37077b9 jenkins-slave:1.7 "/bin/sh -cxe cat ..." 21 seconds ago Starting sad_fermi f15178ca9657 jenkins-slave:1.7 "/bin/sh -cxe cat ..." 16 minutes ago Exited (0) 10 minutes ago wonderful_mclean 257727b1aa9a jenkins-slave:1.7 "/bin/sh -cxe cat ..." 16 minutes ago Exited (0) 10 minutes ago

Need more system information.

• Linux distro and version.
• Docker daemon version.

I'm running it on vsphere .

Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 11 Server Version: v1.3.1-16055-afdab46 Storage Driver: vSphere Integrated Containers v1.3.1-16055-afdab46 Backend Engine VolumeStores: default swap vSphere Integrated Containers v1.3.1-16055-afdab46 Backend Engine: RUNNING VCH CPU limit: 39600 MHz VCH memory limit: 64 GiB VCH CPU usage: 4529 MHz VCH memory usage: 18.79 GiB VMware Product: VMware vCenter Server VMware OS: linux-x64 VMware OS version: 6.0.0 Registry Whitelist Mode: enabled Whitelisted Registries: x18gvichub01a.prd.cloudnow.dbs.com,x18gvichub01b.prd.cloudnow.dbs.com Plugins: Volume: vsphere Network: bridge Swarm: inactive Operating System: linux-x64 OSType: linux-x64 Architecture: x86_64 CPUs: 39600 Total Memory: 64 GiB ID: vSphere Integrated Containers Docker Root Dir: Debug Mode (client): false Debug Mode (server): false Registry: registry.hub.docker.com Experimental: false Live Restore Enabled: false

Another one issue on below flow.

1. No containers templates configured
2. 10 Jobs pending in Queue and waiting for containers.
3. Configured Template with docker once retention strategy
4. 10 containers spin up and 8 finished job and die.
5. 2 were running as its long-running process.
6. one more job added in the queue and it's waiting for long time to spin a new one.
7. when the 2 current running jobs completed then only the new containers coming up.

is this expected one,? there should be new slave spin up when job added to the queue right?

https://vmware.github.io/vic-product/assets/files/html/1.1/vic_app_dev/container_operations.html vsphere does not appear to be a full implementation of the remote docker API. If you manually delete a single offline agent does the container get removed on the backend? What version of yet another docker plugin are you using?

Yes it does, the version im using is "0.1.0-rc46".

@saibalak80 as a temporary workaround you can create a regularly scheduled freestyle job which cleans up dead agents. You would need the groovy plugin installed and run the following system groovy script. https://github.com/samrocketman/jenkins-script-console-scripts/blob/master/cleanup-offline-agents.groovy

I would personally modify it so that it takes into account the age of the agent (i.e. don't delete agents which were just created and make sure the dead agent is at least 5 minutes old) and run it on an infrequent schedule like no less than every 30 minutes. You don't even necessarily need to account for the age of the agent because the docker plugin will just provision new agents after they were deleted.

It's not a perfect workaround but would address infinite resource starvation.

AFAIR i did cleanup somewhere. In retentionStrtategy?...

I don't experience this when using Docker daemon so it might be a difference with how vsphere containers work compared to Docker.