gitlab: Atlantis apply returns `Status cannot transition via \"run\"` when updating commit check status

[fazith27]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

I started exploring Atlantis recently. I found an issue when running atlantis apply. Pipeline at Gitlab end is continuously running even after applying the changes and the MR is updated with the apply successful comment. Provided atlantis logs below for reference. As per the logs, it looks like Atlantis is failing when making the Gitlab api to update the commit status and the process never ends in the backend. FYI, I am using Gitlab and Terragrunt.

Reproduction Steps

• docker run -p 80:4141 -d --pull=always registry.gitlab.com/mytestproj/tf-stack/custom-atlantis server --gitlab-user=xxxxx --gitlab-token=xxxxx --repo-allowlist=gitlab.com/mytestproj/tf-stack --gitlab-webhook-secret=xxxx--repo-config="/home/atlantis/repos.yaml" --silence-no-projects --checkout-strategy=merge
• Added webhook at Gitlab end to post the MR events to atlantis
• Created MR in Gitlab
• Commented atlantis plan and then atlantis apply 3 to 4 times in the same MR with multiple changes
• First apply ran successfully and after that the apply pipeline never ends in Gitlab

Logs

{
  "level":"warn","ts":"2022-11-14T22:18:33.360Z",
  "caller":"events/apply_command_runner.go:204",
  "msg":"unable to update commit status: POST https://gitlab.com/api/v4/projects/mytestproj/tf-stack/statuses/8b9c4d85c3ed817d7222093412dfa37dc612d81c: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via \"run\")}",
  "json":{"repo":"mytestproj/tf-stack","pull":"7"},
  "stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).updateCommitStatus\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:204\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:163\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296"
}

Dockerfile for my custom image

FROM ghcr.io/runatlantis/atlantis
RUN curl --output terragrunt -L https://github.com/gruntwork-io/terragrunt/releases/download/v0.40.1/terragrunt_linux_amd64 && chmod +x terragrunt && mv terragrunt /usr/local/bin/terragrunt
COPY repos.yaml /home/atlantis

Server side-config (repos.yaml)

repos:
- id: "/.*/"
  # apply_requirements: [approved]
  allowed_overrides: [workflow]
  allow_custom_workflows: true
  workflow: terragrunt
  pre_workflow_hooks:
    - run: chmod +x gen-atlantis-yaml.sh && ./gen-atlantis-yaml.sh
  post_workflow_hooks:
    - run: find . -type d -name ".terragrunt-cache" -prune -exec rm -rf {} \;
workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -input=false -out=$PLANFILE
      - run: terragrunt show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply -input=false $PLANFILE

Atlantis yaml

version: 3
parallel_apply: false
parallel_plan: true
automerge: false
workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -out $PLANFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply $PLANFILE
projects:
- dir: env/np/lb
  workflow: terragrunt
- dir: env/p/lb
  workflow: terragrunt
- dir: env/s/lb
  workflow: terragrunt

[nitrocode]

@fazith27 what atlantis version are you using?

[fazith27]

I am using atlantis 0.19.8. I think I figured out when the issue happens. It is happening when we are doing the targeted apply. I just ran atlantis plan which ran plan for three of my projects but only one project actually had a change. So I ran targeted apply atlantis apply -d <path_to_project>. The apply is successfull as I could see the MR is updated with apply output but the Gitlab pipline is not ended. It shows still running with the same error I posted above.

[nitrocode]

It seems like a gitlab specific issue. I'm unsure if they changed their API or if this is an edge case that no one accounted for.

Seems like the API is throwing the error and it's falling into the error block here.

https://github.com/runatlantis/atlantis/blob/5b88c4224ad2935517482659c9919273e8c55575/server/events/apply_command_runner.go#L196-L205

https://github.com/runatlantis/atlantis/blob/b78c4e8ef9e27378e0ec0dbe6d6157b7940dee3b/server/events/commit_status_updater.go#L67

https://github.com/runatlantis/atlantis/blob/b78c4e8ef9e27378e0ec0dbe6d6157b7940dee3b/server/events/vcs/gitlab_client.go#L235

PRs are welcome if you or someone can troubleshoot this.

[Brightside56]

Have similar/same issue and it's very annoying

{"level":"warn","ts":"2024-01-25T18:24:21.242Z","caller":"events/plan_command_runner.go:191","msg":"unable to update commit status: POST https://somegitlab.com/api/v4/projects/xxx/yyy/statuses/1d513a0d019425c7116e42507e35d11670e4f42e: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via \"run\")}","json":{"repo":"xxx/yyy","pull":"304"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:191\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:306\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:365"}

[tweiss-mdm]

Encountering same issue on Atlantis 0.27.1 and Gitlab 16.8

caller":"events/apply_command_runner.go:221","msg":"unable to update commit status: POST

[elft3r]

Hey, I started looking into this issue by debugging the code. It seems like this warning is a symptom of another problem. My current understanding is that this error is retrieved when you set the pipeline status to running when it is already in the state running.

My next step is to figure out why it's not transitioning to either success or failed. I will keep you posted on what I learned.