feat: add graceful terminate option to terminate-agent-hook

[long-wan-ep]

trafficstars

Description

Resolves https://github.com/cattle-ops/terraform-aws-gitlab-runner/issues/1029.

Adds optional functionality to the terminate-agent-hook module for graceful termination, which users can enabled/configure using new input variables. When graceful terminate is enabled, the lambda will give running jobs a chance to finish before the runner instances are terminated. When graceful terminate is disabled, the original behavior of the terminate-agent-hook is used.

Updated .pylintrc to use /usr/local/lib/python3.12/site-packages/, because pylint was not able to find the boto3 dependencies in /usr/local/lib/python3.11/site-packages/. Seems like megalinter is using python 3.12 now.

Migrations required

No

Verification

With graceful terminate enabled

1. Manually deployed a runner with:

runner_worker_graceful_terminate = {
    enabled      = true
    retry_period = 60
    timeout      = 300
}

2. Started a GitLab job using the runner
3. Manually triggered instance refresh in the runner ASG
4. Verified the new graceful termination steps in the lambda runs successfully
5. Verified the existing steps in the lambda runs successfully

With graceful terminate disabled

1. Manually deployed a runner
2. Started a GitLab job using the runner
3. Manually triggered instance refresh in the runner ASG
4. Verified the lambda runs successfully without graceful termination steps

[github-actions[bot]]

Hey @long-wan-ep! 👋

Thank you for your contribution to the project. Please refer to the contribution rules for a quick overview of the process.

Make sure that this PR clearly explains:

• the problem being solved
• the best way a reviewer and you can test your changes

With submitting this PR you confirm that you hold the rights of the code added and agree that it will published under this LICENSE.

The following ChatOps commands are supported:

• /help: notifies a maintainer to help you out

Simply add a comment with the command in the first line. If you need to pass more information, separate it with a blank line from the command.

This message was generated automatically. You are welcome to improve it.

[long-wan-ep]

Currently if graceful terminate is enabled, the runner is highly likely to run into https://github.com/cattle-ops/terraform-aws-gitlab-runner/issues/1062, since graceful terminate delays runner termination during a runner refresh. I think we should tackle that issue first before we implement this feature.

[kayman-mk]

#1062 will be solved via #1102 soon.

[long-wan-ep]

Since https://github.com/cattle-ops/terraform-aws-gitlab-runner/issues/1062 has been resolved, this is no longer blocked.

[kayman-mk]

@long-wan-ep There is another PR ongoing (#1117 ). On a first view it seems to be that their implementation is more robust. Could you have a look please and comment on that?

[long-wan-ep]

@kayman-mk The implementation from #1117 is much simpler and works, let's close this MR in favour of the other solution.