StackOverflow error on index files

[theveriton]

Suddenly, Perform CodeQL Analysis step has started failing on index files step.

github/codeql-action/analyze@v2

I've tried using v3 too with no luck

Also, I've tried increasing thread heap size from 2M all the way to 512M with no luck

E:\actions-runner-ghec\_work\_tool\CodeQL\2.16.2\x64\codeql\codeql.exe database finalize --finalize-dataset --threads=12 --ram=30002 E:\actions-runner-ghec\_work\_temp\codeql_databases\java
  Picked up JAVA_TOOL_OPTIONS: -Xss512m
  Running pre-finalize script E:\actions-runner-ghec\_work\_tool\CodeQL\2.16.2\x64\codeql\java\tools\pre-finalize.cmd in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver.
  [2024-02-20 03:25:52] [build-stderr] Picked up JAVA_TOOL_OPTIONS: -Xss512m
  [2024-02-20 03:25:52] [build-stderr] Scanning for files in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver...
  [2024-02-20 03:25:52] [build-stderr] E:\actions-runner-ghec\_work\_temp\codeql_databases\java: Indexing files in in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver...
  [2024-02-20 03:25:53] [build-stderr] Running command in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver: [E:\actions-runner-ghec\_work\_tool\CodeQL\2.16.2\x64\codeql\xml\tools\index-files.cmd, E:\actions-runner-ghec\_work\_temp\codeql_databases\java\working\files-to-index18386133814325443289.list]
  [2024-02-20 03:25:53] [build-stdout] [2024-02-20 03:25:53] [build-stderr] Picked up JAVA_TOOL_OPTIONS: -Xss512m
  [2024-02-20 03:25:55] [build-stderr] Picked up JAVA_TOOL_OPTIONS: -Xss512m
  [2024-02-20 03:25:55] [build-stderr] Scanning for files in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver...
  [2024-02-20 03:25:56] [build-stderr] E:\actions-runner-ghec\_work\_temp\codeql_databases\java: Indexing files in in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver...
  [2024-02-20 03:25:56] [build-stderr] Running command in E:\actions-runner-ghec\_work\test-test-appserver\test-test-appserver: [E:\actions-runner-ghec\_work\_tool\CodeQL\2.16.2\x64\codeql\properties\tools\index-files.cmd, E:\actions-runner-ghec\_work\_temp\codeql_databases\java\working\files-to-index182820796413[80](https://github.com/testsomeorg/test-test-appserver/actions/runs/7971172665/job/21760290942#step:6:81)0805[88](https://github.com/testsomeorg/test-test-appserver/actions/runs/7971172665/job/21760290942#step:6:89).list]
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] Picked up JAVA_TOOL_OPTIONS: -Xss512m
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] Exception in thread "main" java.lang.StackOverflowError
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$GroupHead.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$Loop.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$GroupTail.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$BranchConn.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$CharProperty.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$Branch.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$GroupHead.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$Loop.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$GroupTail.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$BranchConn.match(Unknown Source)
  [2024-02-20 03:25:56] [build-stdout] [2024-02-20 03:25:56] [build-stderr] 	at java.base/java.util.regex.Pattern$CharProperty.match(Unknown Source)

[aibaars]

Thanks for reporting. Could you include the entire stack trace or at least the bottom part before it gets to the repeating at java.base/java.util.regex.Pattern lines? Hopefully that can tell us where the problem comes from.

[theveriton]

This is the bottom:

 [2024-02-19 02:40:22] [build-stdout] [2024-02-19 02:40:22] [build-stderr] 	at java.base/java.util.regex.Pattern$Branch.match(Unknown Source)
  [2024-02-19 02:40:22] [build-stdout] [2024-02-19 02:40:22] [build-stderr] 	at java.base/java.util.regex.Pattern$GroupHead.match(Unknown Source)
  [2024-02-19 02:40:22] [build-stdout] [2024-02-19 02:40:22] [build-stderr] 	at java.base/java.util.regex.Pattern$Loop.match(Unknown Source)
  Error: 2-19 02:40:22] [build-stdout] [2024-02-19 02:40:22] [ERROR] Spawned process exited abnormally (code 1; tried to run: [E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\properties\tools\index-files.cmd, E:\actions-runner-ghec-2\_work\_temp\codeql_databases\java\working\files-to-index9666766442730007879.list])
  [2024-02-19 02:40:22] [build-stderr] A fatal error occurred: Exit status 1 from command: [E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\properties\tools\index-files.cmd, E:\actions-runner-ghec-2\_work\_temp\codeql_databases\java\working\files-to-index9666766442730007879.list]
  Error: 2-19 02:40:22] [ERROR] Spawned process exited abnormally (code 2; tried to run: [E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\java\tools\pre-finalize.cmd])
  A fatal error occurred: Exit status 2 from command: [E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\java\tools\pre-finalize.cmd]
  Error: Encountered a fatal error while running "E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\codeql.exe database finalize --finalize-dataset --threads=6 --ram=14437 E:\actions-runner-ghec-2\_work\_temp\codeql_databases\java". Exit code was 2 and error was: A fatal error occurred: Exit status 2 from command: [E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, E:\actions-runner-ghec-2\_work\_tool\CodeQL\2.16.2\x64\codeql\java\tools\pre-finalize.cmd]. See the logs for more details.

[aibaars]

I was afraid of that. I suppose a trace for a stack overflow is too long to show the non-repeating start of it. What project are you running on? Is it open source? I just noticed on a local test that I get a similar stacktrace when running CodeQL on https://github.com/apache/hive .

[theveriton]

No, it isn't open source, i've redacted the repo and org info in the logs. It's a self hosted runner, will the logs be anywhere on that?

[aibaars]

@theveriton Does this problem occur all the time or just this once? I spotted a case like this when analysing apache/hive, but it didn't happen in the dozen re-runs I made. This makes it hard to find the cause.

The easiest way to grab the logs is the re-run a workflow with debug logging enabled. That causes the codeql-action to upload a debug artifact. You could also try to locate the "database" folders in the "RUNNER_TEMP" folder on the self-hosted runner. However, these are cleared up by subsequent jobs.

[aibaars]

@theveriton Those links do not work (luckily). The debug artifacts contain logs, source code and other potentially sensitive data. Since this is a closed source repository you probably do not want to share that on a public issue. You can download and unzip the artifact to check the various logs folders for potentially interesting error messages. Alternatively, you can open an Enterprise support ticket and share the artifact privately with GitHub's support engineers.

Another thing to try would be to set the tools: property of the codeql-action/init step to the URL of last month's version of CodeQL. If we can figure out the last working/first failing versions then that would help narrow down the search for the cause of the problem.

[theveriton]

Awesome! It started working with v2.15.5, doesn't work for 2.16.0 and beyond.

[aibaars]

Thanks, glad things are working for you now. Hopefully the team can figure out what happened between 2.15.5 and 2.16.0 .

[aibaars]

It looks like since 2.16.0 CodeQL extracts all *.properties files by default. Do you have a malformed properties file in your repository by any chance? Could you try adding:

env:
  LGTM_INDEX_PROPERTIES_FILES: "false"

to your workflow file and go back to using the 2.16 release?

[theveriton]

That also worked! v2.16.2...we do have a bunch of property files.

[aibaars]

I think the following regular expression may have performance problems when run on inputs that are not really properties files.

We'll look into improving things. Thanks again for reporting this.