terraform-aws-gitlab-runner icon indicating copy to clipboard operation
terraform-aws-gitlab-runner copied to clipboard

Occasionally EC2s are left in a Stopped state

Open cmavromichalis opened this issue 4 years ago • 5 comments
trafficstars

I've noticed that as I update the GitLab runner versions occasionally there are EC2's in a Stopped state. I have to manually delete these when running the terraform as the terraform won't destroy EC2's in a Stopped state. I haven't been able to root cause why they are put in a stopped state or any kind of pattern for why. It seems to just happen randomly and infrequently.

My scenario is the GitLab CI docker-machine runner - one runner agent as described in the README. We update GitLab and GitLab runner's versions once a month. Sometime during the month of use the EC2 is started then put in a Stopped state.

Are there logs I could look at or provide to help root cause this?

cmavromichalis avatar Oct 25 '21 22:10 cmavromichalis

Haven't seen this before. My machines are always terminated. But let me check my configuration.

kayman-mk avatar Oct 28 '21 18:10 kayman-mk

I didn't find any special settings in my configuration. Do you have log files available?

kayman-mk avatar Oct 30 '21 07:10 kayman-mk

I have looked through the EC2 logs but havent found anything that jumped out at me. Are there specific logs that would help?

cmavromichalis avatar Nov 10 '21 18:11 cmavromichalis

We also didn't see this behaviour

npalm avatar Nov 10 '21 22:11 npalm

This will be indirectly solved by #392

kayman-mk avatar Nov 13 '21 10:11 kayman-mk

@cmavromichalis Shouldn't be an issue anymore and can be closed right?

kayman-mk avatar Dec 24 '22 21:12 kayman-mk