Code scanning is waiting for results from CodeQL; CodeQL is stuck

[jyio-va]

trafficstars

Hello there. I seem to have a stuck GitHub Actions workflow.

When I tried to push to GitHub yesterday, I got the following error:

remote: error: GH013: Repository rule violations found for refs/heads/main.
remote: Review all repository rules at https://github.com/REDACTED_ORG/REDACTED_REPO/rules?ref=refs%2Fheads%2Fmain
remote:
remote: - Code scanning is waiting for results from CodeQL for the commit REDACTED_COMMIT.
remote:
To https://github.com/REDACTED_ORG/REDACTED_REPO.git
 ! [remote rejected] main -> main (push declined due to repository rule violations)
error: failed to push some refs to 'https://github.com/REDACTED_ORG/REDACTED_REPO.git'

Checking the workflow status, I saw that the "CodeQL Setup" action completed successfully. So I asked to re-run the "CodeQL Setup" action. It's been a day now, and it's still stuck in the "Queued" state. So I tried to unstick the job by canceling it, but was told "Failed to cancel workflow."

How do we proceed with this?

Thank you.

[hvitved]

Hi. Sorry for the late reply; is this still an issue?

[GroenteLepel]

Hi, not the person reporting this but I'm having the same struggle on our open source project where a contributor is trying to merge a fork pr. I haven't tried rerunning the workflow, so will look into that. It is referencing two commits in the pr that have been merged from main, of which one is a dependabot merge. So it's not like there is a lot of changes happening.

This is the PR I'm referencing: https://github.com/alliander-opensource/transformer-thermal-model/pull/22

But not sure if that is of any help here. I thought it was the GPGSign that was the issue, so I asked them to update those on their previous commits, causing the force pushes.

Update: I cannot find the workflow of this PR in my CodeQL workflow history. I tried temporarily disabling codeQL via the settings of the repo, updating the PR with a new merge from main but still get the error "waiting for results from CodeQL" but now with two different commit hashes.

[GroenteLepel]

I also found this discussion. It gives some tips to switch from basic to advanced configuration. That was the reason why it did not allow me to rerun the action, I had the basic setup.

[jyio-va]

This is still a problem, and now there are 4 workflows in the pipeline -- queued or scheduled -- but nothing seems to be happening.

✅ Scheduled
CodeQL #4: by github-advanced-security bot
main	
 8 hours ago
 1m 21s

✅ Scheduled
CodeQL #3: by github-advanced-security bot
main	
 last week
 1m 4s

✅ Scheduled
CodeQL #2: by github-advanced-security bot
main	
 2 weeks ago
 1m 1s

🟠 CodeQL Setup
CodeQL #1: by github-advanced-security bot
main	
 2 weeks ago
 Queued

[rvermeulen]

Hi @jyio-va,

Thanks for the follow-up. Is it possible to share the Actions logs?

[rvermeulen]

@shaikhul as the first responder from Code Scanning, do you have an idea what may cause Code Scanning to stall?

[shaikhul]

Not sure, did they share the actions log? which repo they were trying with? are they using default setup or advanced setup?

[GroenteLepel]

Just to chime in, I did switch with my repo to advanced, and it's been running properly now ever since. We did not have any forked pr, so I wasn't able to test it in that case in particular.

[jyio-va]

The CodeQL analysis seems to run weekly, and it's still blocked.

✅ Scheduled
CodeQL #8: by github-advanced-security bot
main	
 3 days ago
 1m 24s

✅ Scheduled
CodeQL #7: by github-advanced-security bot
main	
 last week
 1m 9s

✅ Scheduled
CodeQL #6: by github-advanced-security bot
main	
 2 weeks ago
 1m 7s

✅ Scheduled
CodeQL #5: by github-advanced-security bot
main	
 3 weeks ago
 1m 3s

✅ Scheduled
CodeQL #4: by github-advanced-security bot
main	
 last month
 1m 21s

✅ Scheduled
CodeQL #3: by github-advanced-security bot
main	
 last month
 1m 4s

✅ Scheduled
CodeQL #2: by github-advanced-security bot
main	
 last month
 1m 1s

🟠 CodeQL Setup
CodeQL #1: by github-advanced-security bot
main	
 last month
 Queued

Not sure, did they share the actions log? which repo they were trying with? are they using default setup or advanced setup?

Here, let me share the log from the latest CodeQL run. Repo is department-of-veterans-affairs/vista-enhancement-suite (and another with the same issue department-of-veterans-affairs/lighthouse-vista-enhancement-suite).

I did review the repository rules as recommended in the error message, btw, and there is just one ruleset, Evaluate Policy: Require CodeQL

I assume it's the default setup, since I don't remember explicitly making it advanced. Thanks for the tip @GroenteLepel; I'll try it.

[jyio-va]

I was able to push new commits by creating a new branch and merging the updates through the PR workflow. The web UI reported a couple of questionable CodeQL alerts, and I was able to massage the code to make it work. Perhaps this is the prescribed workflow, but the error messages were not helpful at all, and there was no indication that this was the way to make it work.

Since I created a bunch of useless commits debugging this problem, and there was no way to rewrite the main timeline to clean it up (org settings, probably), I decided to scrap the old repo and create a new one.

[rvermeulen]

@jyio-va I'm asking internally about the queued CodeQL Setup workflow. This is the workflow that is called when enabling or updating default setup.

Do you know if code scanning is using self-hosted runners?

[robertbrignull]

One thing worth noting is that "scheduled" here doesn't mean that it's still waiting to run, but rather than it was triggered automatically like a CRON job instead of being in response to a push or a PR. Since those runs have the "✅" and say they completed in around 1 minute, it doesn't seem like those runs are a problem. And since you provided logs for one of the workflows that means that it ran, or am I misunderstanding?

The setup run that says it's been "queued" for over a month is a bit concerning, but it seems like the only one that was affected was the very first run. I've seen this happen when there aren't suitable runners (e.g. self-hosted runners with the right labels). The actions workflow won't start and it won't automatically recover once you add the runners.

It's unfortunate that it failed to start and cancelling it also didn't go perfectly. Are you still seeing new runs get stuck in the "queued" state?

When I tried to push to GitHub yesterday, I got the following error:

Just checking but this happens when you try to push directly to a branch, and not using a PR? Could you confirm what your branch protection rules / rulesets are?

I was wondering if there's a rule requiring a CI check to have passed before a commit is pushed, which would then block all pushes because CI can't start until after the push.

[jyio-va]

Yes, I believe these scheduled jobs actually ran.

This happens when I try to push directly to the main (default) branch, not any other branch. Branch protection rules are disabled for the repo (since I'm the only author at this time). Perhaps the org has rules that override the repo rules? I don't know how to check that.

[garnertb]

@jyio-va

There is a GitHub team that supports your organization directly and has more details on org-specific configuration and policies. If this is still a problem for you, consider creating an issue in the org’s support repository where you can include direct links to the repos/logs for the team to investigate. If you do that, you can close this issue, but please provide a link to it in the new issue. I also recommend reviewing the code scanning policy in your organization’s GitHub handbook as it documents many of the policies and configuration being applied.

[jyio-va]

Okay, I see that the VA has configured Required Reviews for Pull Requests at the organization level. The documentation suggests this setting requires that changes to protected branches be pushed via pull requests.

Does it also stipulate that the default branch be protected? Does CodeQL do this? What other settings could have this effect? Thank you for helping me investigate.