atlantis
atlantis copied to clipboard
runatlantis
runtime error: invalid memory address or nil pointer dereference
Hi,
I am trying to integrate atlantis (v0.17.4) with gitlab (free self managed ver. 11.9.6).
It is working fine till autoplan
Comment output:Ran Plan for project: preprod dir: environments/preprod workspace: default
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# module.eks_1.null_resource.test0 will be created
+ resource "null_resource" "test0" {
+ id = (known after apply)
}
# module.eks_1.null_resource.test1 will be created
+ resource "null_resource" "test1" {
+ id = (known after apply)
}
Plan: 2 to add, 0 to change, 0 to destroy.
-
:arrow_forward: To apply this plan, comment:
-
atlantis apply -p preprod
-
-
:put_litter_in_its_place: To delete this plan click ***
-
:repeat: To plan this project again, comment:
-
atlantis plan -p preprod
-
-
:fast_forward: To apply all unapplied plans from this pull request, comment:
-
atlantis apply
-
-
:put_litter_in_its_place: To delete all plans and locks for the PR, comment:
-
atlantis unlock
-
But fails when I do "atlantis apply -p preprod": Error: goroutine panic. This is a bug.
runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:221 (0x449106)
/usr/local/go/src/runtime/signal_unix.go:735 (0x4490d6)
/home/circleci/project/server/events/vcs/gitlab_client.go:208 (0x96c66a)
/home/circleci/project/server/events/vcs/proxy.go:72 (0x96dbd4)
/home/circleci/project/server/events/apply_command_runner.go:101 (0xe436b4)
/home/circleci/project/server/events/command_runner.go:214 (0xe476a3)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x463f40)
Following is the request body sent by gitlab
{
"object_kind": "note",
"event_type": "note",
"user": {
"name": "***",
"username": "***",
"avatar_url": "***/uploads/-/system/user/avatar/503/avatar.png"
},
"project_id": 953,
"project": {
"id": 953,
"name": "terraform",
"description": "",
"web_url": "***",
"avatar_url": null,
"git_ssh_url": "***",
"git_http_url": "***",
"namespace": "***",
"visibility_level": 0,
"path_with_namespace": "***",
"default_branch": "master",
"ci_config_path": null,
"homepage": "***",
"url": "***",
"ssh_url": "***",
"http_url": "***"
},
"object_attributes": {
"attachment": null,
"author_id": 503,
"change_position": null,
"commit_id": null,
"created_at": "2021-10-12 15:07:52 UTC",
"discussion_id": "82fd2f394e41ef45062f5b20021d35caa8eea8af",
"id": 117876,
"line_code": null,
"note": "atlantis apply -p preprod",
"noteable_id": 33736,
"noteable_type": "MergeRequest",
"original_position": null,
"position": null,
"project_id": 953,
"resolved_at": null,
"resolved_by_id": null,
"resolved_by_push": null,
"st_diff": null,
"system": false,
"type": null,
"updated_at": "2021-10-12 15:07:52 UTC",
"updated_by_id": null,
"description": "atlantis apply -p preprod",
"url": "***/terraform/merge_requests/7#note_117876"
},
"repository": {
"name": "terraform",
"url": "***",
"description": "",
"homepage": "***/terraform"
},
"merge_request": {
"assignee_id": null,
"author_id": 503,
"created_at": "2021-10-12 15:06:48 UTC",
"description": "",
"head_pipeline_id": 15358,
"id": 33736,
"iid": 7,
"last_edited_at": null,
"last_edited_by_id": null,
"merge_commit_sha": null,
"merge_error": null,
"merge_params": {
"force_remove_source_branch": "0"
},
"merge_status": "can_be_merged",
"merge_user_id": null,
"merge_when_pipeline_succeeds": false,
"milestone_id": null,
"source_branch": "div-fb",
"source_project_id": 953,
"state": "opened",
"target_branch": "master",
"target_project_id": 953,
"time_estimate": 0,
"title": "check1",
"updated_at": "2021-10-12 15:07:52 UTC",
"updated_by_id": null,
"url": "***/terraform/merge_requests/7",
"source": {
"id": 953,
"name": "terraform",
"description": "",
"web_url": "***/terraform",
"avatar_url": null,
"git_ssh_url": "***",
"git_http_url": "***",
"namespace": "***",
"visibility_level": 0,
"path_with_namespace": "***/terraform",
"default_branch": "master",
"ci_config_path": null,
"homepage": "***/terraform",
"url": "***/terraform.git",
"ssh_url": "***/terraform.git",
"http_url": "***"
},
"target": {
"id": 953,
"name": "terraform",
"description": "",
"web_url": "***/terraform",
"avatar_url": null,
"git_ssh_url": "***/terraform.git",
"git_http_url": "***/terraform.git",
"namespace": "***",
"visibility_level": 0,
"path_with_namespace": "***/terraform",
"default_branch": "master",
"ci_config_path": null,
"homepage": "***/terraform",
"url": "***/terraform.git",
"ssh_url": "***/terraform.git",
"http_url": "***/terraform.git"
},
"last_commit": {
"id": "5facef328360f37b26da3268d9cdc438fcc30cd4",
"message": "check1\n",
"timestamp": "2021-10-12T15:02:49Z",
"url": "***/terraform/commit/5facef328360f37b26da3268d9cdc438fcc30cd4",
"author": {
"name": "***",
"email": "***"
}
},
"work_in_progress": false,
"total_time_spent": 0,
"human_total_time_spent": null,
"human_time_estimate": null
}
}
Installation repoConfig
repoConfig: |
---
repos:
- id: ***/terraform
# apply_requirements: [mergeable]
allowed_overrides: [apply_requirements, workflow, delete_source_branch_on_merge]
allowed_workflows: [preprod_workflow]
delete_source_branch_on_merge: true
# workflows lists server-side custom workflows
workflows:
preprod_workflow:
plan:
steps:
- init:
extra_args: ["-input=false", "-no-color"]
- plan:
extra_args: ["-input=false", "-no-color", "-refresh=false", "-parallelism=30"]
apply:
steps:
- apply:
extra_args: ["-no-color", "-input=false", "-parallelism=30"]
atlantis.yaml at repo's root
version: 3
automerge: true
projects:
- name: preprod
dir: ./environments/preprod
workspace: default
terraform_version: v1.0.2
autoplan:
when_modified: ["./*.tf"]
enabled: true
workflow: preprod_workflow
Thanks for any help!
I believe this is the same issue I am having, which is reported here: #1804
This is really disrupting our workflow and it is unclear what we can do about it.
I have had the same issue. #1931 doesn't really fix it... updating didn't change anything.
I have had the same issue. #1931 doesn't really fix it... updating didn't change anything.
@marceloboeira Hmm is it the same stack trace? There could be another nil pointer hiding somewhere that I can't reproduce.
Yeap:
runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:221 (0x449106)
runtime/signal_unix.go:735 (0x4490d6)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 (0x98192a)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0x982eb4)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0x983654)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xe505f4)
github.com/runatlantis/atlantis/server/events/command_runner.go:252 (0xe54a03)
runtime/asm_amd64.s:1581 (0x463fa0)
@sapslaj I realised that this started to happen after we've updated gitlab recently, but interestingly enough it doesn't happen to all repos with terraform we have.
Ahh yes. I was testing with GitHub instead of GitLab so this might be a different issue unrelated to what I fixed then. My bad for linking my PR to this issue without taking a closer look.
It looks like it's happening somewhere in PullIsMergable. I wonder if something changed with the GitLab update? It looks like Atlantis is on the latest go-gitlab version at least.
Yes, so we kind of found a way to overpass this bug by creating an empty pipeline on the repo. It seems that on line 208:
statuses, _, err := g.Client.Commits.GetCommitStatuses(mr.ProjectID, mr.HeadPipeline.SHA, nil)
the mr.HeadPipeline could be empty, leading to the invalid memory address or nil pointer dereference, since the .SHA property is expected.
I didn't have much time to check what could be an alternative here, maybe using pull.HeadCommit instead...
However, it could be other dependencies of mr.HeadPipeline like line 221 also have that issue...
isPipelineSkipped := mr.HeadPipeline.Status == "skipped"
A more complete log + stack trace overview:
{"level":"info","ts":"2022-01-06T14:05:29.474Z","caller":"events/events_controller.go:417","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"monitors\" flags=\"\"","json":{}}
{"level":"warn","ts":"2022-01-06T14:05:30.069Z","caller":"events/apply_command_runner.go:97","msg":"unable to update commit status: POST $$$REDACTED$$$: 403 {message: 403 Forbidden}","json":{"repo":"%%","pull":"$$"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:97\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:252"}
{"level":"error","ts":"2022-01-06T14:05:30.965Z","caller":"events/command_runner.go:377","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:221 (0x449106)\nruntime/signal_unix.go:735 (0x4490d6)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 (0x98192a)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0x982eb4)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0x983654)\ngithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xe505f4)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:252 (0xe54a03)\nruntime/asm_amd64.s:1581 (0x463fa0)\n","json":{"repo":" $$$REDACTED$$$","pull":" $$$REDACTED$$$"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:377\nruntime.gopanic\n\truntime/panic.go:1038\nruntime.panicmem\n\truntime/panic.go:221\nruntime.sigpanic\n\truntime/signal_unix.go:735\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:105\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:252"}
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:97
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
github.com/runatlantis/atlantis/server/events/command_runner.go:252
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics
github.com/runatlantis/atlantis/server/events/command_runner.go:377
runtime.gopanic
runtime/panic.go:1038
runtime.panicmem
runtime/panic.go:221
runtime.sigpanic
runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
github.com/runatlantis/atlantis/server/events/command_runner.go:252
After a lot of debugging, it seems that the root cause is a broken permission token + gitlab updating.
In resume, we have a user for Atlantis with a token created a long time ago. Over gitlab updates, this token doesn't work the same way a new token, created with the same permissions.
https://github.com/runatlantis/atlantis/blob/55f5d8171b3b852ccee6067504a5530157ba5154/server/events/vcs/gitlab_client.go#L195-L199
This code doesn't error, yet it results on a broken mr struct.
The HeadPipeline is nil, which makes a problem for mr.HeadPipeline.SHA, I believe it is caused by the lack of permissions for the token to create/run the pipeline (might be the feature permissions changed but the token didn't?)
https://github.com/runatlantis/atlantis/blob/55f5d8171b3b852ccee6067504a5530157ba5154/server/events/vcs/gitlab_client.go#L208-L211
Running locally, I could simply replace the code to run pull.HeadCommit and it has the same effect, however, as mentioned before, the other line using HeadPipeline fails:
https://github.com/runatlantis/atlantis/blob/55f5d8171b3b852ccee6067504a5530157ba5154/server/events/vcs/gitlab_client.go#L221
runtime error: invalid memory address or nil pointer dereference
/usr/local/Cellar/go/1.17.1/libexec/src/runtime/panic.go:221 (0x1048d26)
panicmem: panic(memoryError)
/usr/local/Cellar/go/1.17.1/libexec/src/runtime/signal_unix.go:735 (0x1048cf6)
sigpanic: panicmem()
/atlantis/server/events/vcs/gitlab_client.go:221 (0x15805d7)
(*GitlabClient).PullIsMergeable: isPipelineSkipped := mr.HeadPipeline.Status == "skipped"
/atlantis/server/events/vcs/proxy.go:72 (0x1581994)
(*ClientProxy).PullIsMergeable: return d.clients[repo.VCSHost.Type].PullIsMergeable(repo, pull)
/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0x1582134)
(*pullReqStatusFetcher).FetchPullStatus: mergeable, err := f.client.PullIsMergeable(repo, pull)
/atlantis/server/events/apply_command_runner.go:105 (0x1a4e6d4)
(*ApplyCommandRunner).Run: ctx.PullRequestStatus, err = a.pullReqStatusFetcher.FetchPullStatus(baseRepo, pull)
/atlantis/server/events/command_runner.go:252 (0x1a52ae3)
(*DefaultCommandRunner).RunCommentCommand: cmdRunner.Run(ctx, cmd)
/usr/local/Cellar/go/1.17.1/libexec/src/runtime/asm_amd64.s:1581 (0x1063e60)
goexit: BYTE $0x90 // NOP
In this case, the pipeline is skipped so we could simply:
isPipelineSkipped := mr.HeadPipeline == nil || mr.HeadPipeline.Status == "skipped"
I have opened #1981 as a way to go around this and avoid the runtime panic...
For people coming into this issue, if the stack trace is the same, you might've updated GitLab recently which is one of the possible causes for this issue.
Solution: create a new token for atlantis with the api permissions and try that. It should go around this issue!
The error on atlantis:
runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:221 (0x449366)
runtime/signal_unix.go:735 (0x449336)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 (0x98598a)
after some troubleshooting using the same gitlab project, branch and commit id with two separate merge request i got two distinct results calling the GitLab API: https://gitlab.com/api/v4/projects/:id/merge_requests/:iid for the field that is used by the go-gitlab module to retrieve the value for mr.HeadPipeline
...
"first_deployed_to_production_at": null,
"pipeline": null,
"head_pipeline": null,
"diff_refs": {
"base_sha": "844 ...
and
"first_deployed_to_production_at": null,
"pipeline": {
"id": 47 ...
},
"head_pipeline": {
"id": 47 ...
Which means there is not issue with the token used because i'm using the same one and i got the two results, one with data on head_pipeline field and the other one with null (that causes the panic error on atlantis)
This discards that there is an issue on the module used to access the Gitlab API (github.com/xanzy/go-gitlab) that is doing the expected work or issues on permission access caused by the token.
IF the row 208 requires the SHA of the commit for the head_pipeline, maybe that value can be found under diff_refs property head_sha
I observed the same issue with our gitlab instance after updating atlantis to 0.19.2. I downgraded back to 0.17.4, the version we had before the update, and everything currently works again. So while its almost definitely an issue with the gitlab api, older versions do seem to be fine with the api issue.
@jamengual yes, the patch I wrote wasn't accepted so if it wasn't fixed in the gitlab client it can (and does) still happen. We recently updated gitlab on our side and started getting this error again.
https://github.com/runatlantis/atlantis/pull/1981
In my case, the problem disappeared after changing the role of the access token from "reporter" to "developer". I don't know if it has anything to do with the error, but I hope it helps someone!
@c0da for us it was always developer, but somehow we had to renew tokens for every gitlab update...
This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
following this workaround helped me out https://github.com/runatlantis/atlantis/issues/2606#issuecomment-1300908417
$ cat .gitlab-ci.yml
dummy:
script:
- echo "dummy job for atlantis to trigger pipelineID"
I'm running into the same issue using atlantis v0.23.3, and gitlab version 15.9. I've rotated access tokens, removed the .terraform.lock.hcl file, attempted the gitlab-ci.yml workaround - all with no success. Here is the stack trace I'm receiving:
runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44e07c)
runtime/signal_unix.go:837 (0x44e04c)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:225 (0xd2d9a6)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xd33184)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xd33a55)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xf3f6b4)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xf44d43)
runtime/asm_amd64.s:1598 (0x46b220)
Any other suggestions?
it looks like as @marceloboeira said even updating the library did not fixed anything and it broke more than just this.....we will evaluate if this + other fixes are needed and report back but we currently do not have anyone actively working on this.
@jamengual I used to have a "safety check" PR open for this: https://github.com/runatlantis/atlantis/pull/1981
Let me know if it makes sense to rebase/reopen that one.
This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'
From what I've gathered, Atlantis is not compatible with GitLab until gitlab fixes their API response.
This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'
Seeing the same issue on Github enterprise server 3.8.x. In my case looks like it happens if the PR does not meet the mergable requirements and the user trying to run atlantis apply:
runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44eb55)
runtime/signal_unix.go:835 (0x44eb25)
github.com/runatlantis/atlantis/server/events/vcs/github_client.go:431 (0xcd322b)
github.com/runatlantis/atlantis/server/events/vcs/github_client.go:500 (0xcd37f7)
github.com/runatlantis/atlantis/server/events/vcs/instrumented_client.go:185 (0xcda60b)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xcdc2a4)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xcdcb75)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xece554)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xed36a3)
runtime/asm_amd64.s:1594 (0x469ac0)
Is there a timeline when this will be fixed in the gitlab-client api of Atlantis?
@Mazorius
There is no timeline. This project is based on community contributed PRs and it's been decided that we cannot merge #3428 without tests. If you'd like to contribute unit tests to the existing pr 3428, that would be the most helpful way to accelerate fixing this issue.
fwiw, i ran into this issue on atlantis 0.24.4 on Gitlab SaaS: 16.3.0-pre 2719a0163a1
I'm not sure precisely what fixed it -- maybe all the things! I changed the three things I found in this thread:
- added the dummy gitlab-ci job (i previously had no pipelines defined)
- removed the hcl locks
-
granted the access token
Developerlevel access
I'm back on the road for now. Thank you to all who shared their experiences.
