codeql
codeql copied to clipboard
github
Extraction failing for go repo
Description of the issue We are running into an issue for a go repo (https://github.com/Azure/ARO-RP) where the extraction has started failing very often for both codeQL v1 and v2. The error message we are seeing in the logs is the following (for v1) (v1 and v2 failure logs are the same actually):
2022-07-22T13:19:39.3311218Z [2022-07-22 13:19:34] [build-stderr] 2022/07/22 13:19:34 Extraction failed: signal: killed
2022-07-22T13:19:39.3311997Z [2022-07-22 13:19:34] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/tools/linux64/preload_tracer, /opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/go/tools/autobuild.sh])
2022-07-22T13:19:39.3312741Z [2022-07-22 13:19:34] Exception caught at top level: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/go/tools/autobuild.sh]
2022-07-22T13:19:39.3313336Z com.semmle.cli2.database.DatabaseProcessCommandCommon.executeSubcommand(DatabaseProcessCommandCommon.java:215)
2022-07-22T13:19:39.3313942Z com.semmle.cli2.database.TraceCommandCommand.executeSubcommand(TraceCommandCommand.java:86)
2022-07-22T13:19:39.3314440Z com.semmle.cli2.picocli.SubcommandCommon.call(SubcommandCommon.java:500)
2022-07-22T13:19:39.3314880Z com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:205)
2022-07-22T13:19:39.3315330Z com.semmle.cli2.picocli.SubcommandMaker.runMain(SubcommandMaker.java:214)
2022-07-22T13:19:39.3315698Z com.semmle.cli2.CodeQL.main(CodeQL.java:98)
and this is the one we get with v2:
[2022-07-22 13:18:44] [build-stderr] 2022/07/22 13:18:44 Extracting /home/runner/work/ARO-RP/ARO-RP/pkg/operator/controllers/dnsmasq/machineconfig_controller.go
[2022-07-22 13:19:34] [build-stderr] 2022/07/22 13:19:34 Extraction failed: signal: killed
Error: 7-22 13:19:34] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/tools/linux64/preload_tracer, /opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/go/tools/autobuild.sh])
A fatal error occurred: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/go/tools/autobuild.sh]
Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/codeql' failed with exit code 2
Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/codeql' failed with exit code 2
at toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v1/lib/toolrunner-error-catcher.js:86:19)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async Object.extractScannedLanguage (/home/runner/work/_actions/github/codeql-action/v1/lib/codeql.js:548:13)
at async createdDBForScannedLanguages (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:82:13)
at async finalizeDatabaseCreation (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:101:5)
at async runFinalize (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:249:5)
at async run (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:83:9)
at async runWrapper (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:221:9)
##[debug]Sending status report: {"workflow_run_id":2718719647,"workflow_name":"CodeQL","job_name":"analyze","analysis_key":".github/workflows/codeql-analysis.yml:analyze","commit_oid":"be4f5c6dfe004206be1b65f42a04c21246376088","ref":"refs/pull/2279/merge","action_name":"finish","action_ref":"v1","action_oid":"unknown","started_at":"2022-07-22T12:56:13.455Z","action_started_at":"2022-07-22T12:56:23.452Z","status":"failure","runner_os":"Linux","action_version":"1.1.16","cause":"The process '/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/codeql' failed with exit code 2","exception":"Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/codeql' failed with exit code 2\n at toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v1/lib/toolrunner-error-catcher.js:86:19)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:97:5)\n at async Object.extractScannedLanguage (/home/runner/work/_actions/github/codeql-action/v1/lib/codeql.js:548:13)\n at async createdDBForScannedLanguages (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:82:13)\n at async finalizeDatabaseCreation (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:101:5)\n at async runFinalize (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze.js:249:5)\n at async run (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:83:9)\n at async runWrapper (/home/runner/work/_actions/github/codeql-action/v1/lib/analyze-action.js:221:9)","completed_at":"2022-07-22T13:19:35.370Z","matrix_vars":"{\n \"language\": \"go\"\n}","runner_arch":"X64","codeql_version":"2.10.0","ml_powered_javascript_queries":"false"}
/opt/hostedtoolcache/CodeQL/0.0.0-20220623/x64/codeql/codeql database bundle /home/runner/work/_temp/codeql_databases/go --output=/home/runner/work/_temp/codeql_databases/db-go.zip --name=db-go
The failure happens for apparently random files every time it happens and we are not able to get more information turning on the debug logging on github actions.
The repo is relatively big with a bunch of dependencies, would that be a factor in these failures ?
If you need access to the repo, we can maybe arrange something
Worked for me: Halving the concurrency the Go extractor attempts using
env:
CODEQL_EXTRACTOR_GO_MAX_GOROUTINES: 16
(default value is 32)
We're seeing the same thing in our CodeQL runs on https://github.com/gravitational/teleport.
From https://github.com/gravitational/teleport/runs/7904788692?check_suite_focus=true
[2022-08-18 19:05:57] [build-stderr] 2022/08/18 19:05:57 Extracting /home/runner/work/teleport/teleport/tool/tctl/common/user_command.go
[2022-08-18 19:06:04] [build-stderr] 2022/08/18 19:06:04 Extraction failed: signal: killed
Error: 8-18 19:06:04] [ERROR] Spawned process exited abnormally (code 1; tried to run: [/opt/hostedtoolcache/CodeQL/0.0.0-20220728/x64/codeql/tools/linux64/preload_tracer, /opt/hostedtoolcache/CodeQL/0.0.0-20220728/x64/codeql/go/tools/autobuild.sh])
A fatal error occurred: Exit status 1 from command: [/opt/hostedtoolcache/CodeQL/0.0.0-20220728/x64/codeql/go/tools/autobuild.sh]
Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20220728/x64/codeql/codeql' failed with exit code 2
Error: The process '/opt/hostedtoolcache/CodeQL/0.0.0-20220728/x64/codeql/codeql' failed with exit code 2
at toolrunnerErrorCatcher (/home/runner/work/_actions/github/codeql-action/v2/lib/toolrunner-error-catcher.js:86:19)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Object.extractScannedLanguage (/home/runner/work/_actions/github/codeql-action/v2/lib/codeql.js:561:13)
at async createdDBForScannedLanguages (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:82:13)
at async finalizeDatabaseCreation (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:101:5)
at async runFinalize (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze.js:249:5)
at async run (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:83:9)
at async runWrapper (/home/runner/work/_actions/github/codeql-action/v2/lib/analyze-action.js:221:9)
Hey , sorry for the late answer , but yes it seemed to have fixed it, thank you !
@reedloden have you tried the goroutine-limiting suggestion above?
Yup, and it's still occurring. See https://github.com/gravitational/teleport/runs/7946318831?check_suite_focus=true
Try taking it down again to 8?
Probably unrelated to your trouble: I note your code uses Go 1.18, so you should probably use the setup-go routine to install it, as the default on Actions workers is still 1.17.