lifecycle hook to wait until all terraform processes are completed

[nitrocode]

Lifecycle hooks are supported but it would be nice to add

• either a default lifecycle to prevent terraform from being killed in the middle of an apply or plan
  • otherwise we could have resources created and the information is not saved into the tfstate
  • otherwise we could have stale dynamodb locks
• or document the lifecycle hook to be optionally added

cc: @Balsir since you added the lifecycle feature to this repo, have you figured out a good preStop lifecycle ?

References

• https://github.com/runatlantis/helm-charts/pull/220
• https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/

[Balsir]

hi @nitrocode, unfortunately haven't had the time for it yet.

The initial idea was to trap SIGTERM via preStop and wait for a configurable amount of time to attempt to let Terraform finish. Atlantis could then proceed with graceful termination (https://github.com/runatlantis/atlantis/pull/1051).

A little food for thought: since the server already handles graceful termination to an extent, wouldn't it be better to implement the use case on the server rather than in the Helm chart?

[nitrocode]

Yes, graceful termination should be implemented in the server. I was thinking this lifecycle method could be a nice workaround for now. :smile:

[GenPage]

@nitrocode Should we close this issue and create one on the Atlantis repo?

[nitrocode]

I think we can have both issues. One here for lifecycle method and one for graceful termination in the server.

[mubarak-j]

This is what I came up with, let me know if seems okay and I will create a PR.

statefulSet:
  terminationGracePeriodSeconds: 180

lifecycle:
  preStop:
    exec:
      command:
        - /bin/bash
        - -c
        - while [ "$(pgrep -f terraform)" ]; do sleep 1; done