GitLab: pipeline is disconnected from commit status

[humitos]

trafficstars

I'm not 100% sure how pipelines work on GitLab, so there may be some mistakes on this issue.

We had a customer report saying that the commit status gets disconnected from a GitLab pipeline, which makes the Merge Request required checks to fail since Read the Docs check is not properly assigned to that pipeline; making the MR impossible to be merged.

After asking for more information about the issue, and asking the GitLab support directly, it seems that our API call must send ref= if we want to associate that commit status with a particular pipeline.

If the commit is associated with a merge request, the API call must target the commit in the merge request’s source branch.

(from https://docs.gitlab.com/ee/api/commits.html#set-the-pipeline-status-of-a-commit)

However, I didn't find an example in the documentation that sends ref= to understand what is the syntax of the value. Reading the response from https://docs.gitlab.com/ee/api/commits.html#commit-status, I think we just need to send ref=<branch> but I'm not sure.

Work required

1. Create an example GitLab repository that uses pipelines
2. Import the project on Read the Docs
3. Reproduce the issue by finding the pipeline disconnection
4. Send ref=<branch> in the API call
5. Check if that connects the pipeline correctly

Context

• Code that hits the GitLab API: https://github.com/readthedocs/readthedocs.org/blob/bc8a201f365c3c1cb1c4c4194d74bdec7e0b591d/readthedocs/oauth/services/gitlab.py#L545-L551
• Support request sent by a customer: https://app.frontapp.com/open/msg_1gv44lhj?key=6SN8Grzuen0BIC0TC586coiaqCxqsxHj

Front conversations

[sstrong99]

a couple comments:

which makes the Merge Request required checks to fail since Read the Docs check is not properly assigned to that pipeline; making the MR impossible to be merged.

This isn’t true. The result is that the MR will only check for the completion of one of the two pipelines, not both. So some required checks might be missed if you don’t realize what happened.

I think we just need to send ref= but I'm not sure.

While the gitlab documentation does make it seem like that’s the correct path, according to gitlab support it is not. A gitlab pipeline can be associated with a commit, a branch, an MR, or a tag (maybe more?). Since the readthedocs builds are meant to be associated with an MR (the readthedocs setting that triggers the behavior is "build pull requests for this project"), the readthedocs pipeline should ensure that when it reports its status it is doing it for a pipeline associated with an MR, not any of the other things. In order to achieve that the gitlab support people said that you can pass “ref=refs/merge-requests//head”, where is the merge request ID (an integer) that you receive in the webhook that triggers the build. I realize that this is not specified in the documentation.

[sstrong99]

Hi @humitos - any updates on this? Thanks! -Steve

[humitos]

Hi @sstrong99. I hadn't had the change to jump into this yet unfortunately.

[agjohnson]

I am doing some tangential work on our new dashboard around pull request builds and gave a quick try to this. In what I am assuming to be a very basic test case -- add a commit in a new merge request and trigger a single pipeline -- the pipeline seems to be connected at the start of the RTD build and at the end:

But this makes sense to me as the issue seems to be when there are multiple pipelines on the repository. Just noting this for additional testing we do.

[agjohnson]

Actually, maybe I just replicated this? My build might be in a weird state as I have rebuilt the versions dozens of times at this point.

But I have two pipelines, from two separate commits.

And the merge request details shows it waiting on the pipeline status:

@sstrong99 is this similar to what you notice?

[agjohnson]

And it seems this is maybe not the ref to use still, as I tried refs/merge-requests/3/head and the pipeline is disconnected from the MR now:

[sstrong99]

That is not exactly what i see, but maybe related? To see what i see, you would also need a separate pipeline defined in a .gitlab-ci.yml file. Often (always?) the first pipeline to run on a given MR works fine, in that a single pipeline is created, which contains the steps defined in .gitlab-ci.yml, and also contains the readthedocs step. After that, when commits are made to the MR, the pipeline gets retriggered, and often the readthedoc step ends up in a seperate pipeline than the steps defined in .gitlab-ci.yml. This is problematic, because when deciding if the ci/cd passed, the MR only cares if the later of those two pipelines passes, and it ignores the other one. But the MR always chooses one of the two pipelines to care about, i've never seen it just hang in the "checking pipeline status" state.

[agjohnson]

Okay, so setting up defined jobs, I am still seeing the same flow that I was, and this doesn't quite match up with what you are saying.

Namely, if I push a second commit to my MR with defined jobs, the first pipeline does not get retriggered, but instead a new pipeline starts. The RTD job is still a separate step on the pipeline:

@sstrong99 perhaps do you have a GitLab config that would could help reproduce this issue? A minimal configuration helps, it seems like there is something specific with your pipeline configuration.

[sstrong99]

if I push a second commit to my MR with defined jobs, the first pipeline does not get retriggered, but instead a new pipeline starts.

Yes, that's the correct behavior, and the same as what I see, except that sometimes two new pipelines start, instead of just one. Just one is the correct behavior.

Here are a couple screen shots of a current MR. The pipeline defined in .gitlab-ci.yml has 2 stages, and the readthedocs build should be the third stage. You can see that some of the pipelines have only 2 stages associated with them, meaning that the readthedocs build is not associated. For those ones, you can find the readthedocs pipeline (which only has one stage) farther down the list, by matching the commit message.

we're using this template for this specific pipeline in our .gitlab-ci.yml:

include:
  # this template makes pipelines run only for merge requests, default branch, and tags
  - template: "Workflows/MergeRequest-Pipelines.gitlab-ci.yml"

although in other repositories where we also observe the behavior we have this instead

# this makes pipelines run only for merge requests and default branch
workflow:
  rules:
    - if: $CI_MERGE_REQUEST_IID
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH