codeql-action
codeql-action copied to clipboard
[proposed enhancement] Error: File size (2838487234) is greater than possible Buffer:
Hi @aeisenberg , @adityasharad ,
As requested here it the workflow run: https://github.com/apache/ofbiz-framework/actions/runs/1470420767
TIA for you help, much appreciated.
Thanks. We'll take a look at the SARIF file from that run and get back to you.
Hi @aeisenberg , @adityasharad ,
Did you get a chance to have a look?
Hi @aeisenberg , @adityasharad ,
Any news?
Hi @aeisenberg , @adityasharad ,
Is Apache OFBiz the only project crossing this issue?
Hi @JacquesLeRoux, thank you for your patience and apologies for the time it's taken to get back to you.
I've had a look at the SARIF file from the Java analysis on your workflow. There are two problems here:
- This results file is too large for the CodeQL Action to process in-memory (hence the Node.js error you see).
- Even if we managed to load it into memory and upload it, it would be rejected by the code scanning service because of the large number of data flow path locations within the analysis results.
Here is an explanation of what I looked for in the SARIF file:
- Total number of alerts: 186
-
jq '.runs[0].results | length' java.sarif
- The current limit supported by code scanning is 5000 per run, so this is not a problem.
- This is not a particularly high number.
-
- Number of flow paths per alert:
-
jq '.runs[0].results | map({query_id: .rule.id, numPaths: .codeFlows | length})' java.sarif
- Some of the results have 732 or more paths each. This is a potential cause of the large results file.
-
- Number of flow path locations per result:
-
jq '.runs[0].results | map({query_id: .rule.id, locations: .locations | length, numPaths: .codeFlows | length, numPathLocations: (if .codeFlows then .codeFlows | map(.threadFlows[0].locations | length) else [] end) | add //0})' java.sarif
- This is likely the problem. Some of the results here have over 20,000 total locations within their data flow paths, which makes the JSON very large. The code scanning service will also reject results with over 5000 such locations per result.
-
Why are there so many paths and path locations?
- This is an unfortunate effect of the way CodeQL groups alerts that have the same location. I suspect we have many alerts here that describe the same vulnerability, with multiple possible sources of tainted data, all ending in the same "sink". CodeQL only produces 4 paths per alert at first, but then it groups those alerts together if they have the same sink location.
- So for example 183 alerts, each with the same sink and 4 paths, get grouped into 1 alert with 732 paths.
How can we make your runs succeed?
- I'm discussing this with my colleagues so that we can address it properly in CodeQL. This requires some careful design to avoid breaking existing alerts, so it's unfortunately not a fix I can promise in the short term.
- Short term workaround: disable the analysis for Java, and run it only for JavaScript while we investigate further.
- Short term workaround: keep the analysis for Java, but disable the queries that are producing more than 5000 path locations per alert. See the summary below for the query IDs that are producing a large number of paths, and this example on how to disable specific queries.
I appreciate this is not an ideal solution, but I hope it's enough to keep your code scanning workflows from failing while we think of a more principled solution.
Summary of SARIF alerts, with path and path location counts per alert
[
{
"query_id": "java/stack-trace-exposure",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/stack-trace-exposure",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/stack-trace-exposure",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/stack-trace-exposure",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/stack-trace-exposure",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/stack-trace-exposure",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/groovy-injection",
"numPaths": 764,
"numPathLocations": 35282
},
{
"query_id": "java/groovy-injection",
"numPaths": 768,
"numPathLocations": 53374
},
{
"query_id": "java/http-response-splitting",
"numPaths": 761,
"numPathLocations": 27411
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 28355
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 28355
},
{
"query_id": "java/http-response-splitting",
"numPaths": 1,
"numPathLocations": 4
},
{
"query_id": "java/http-response-splitting",
"numPaths": 1,
"numPathLocations": 4
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 31965
},
{
"query_id": "java/http-response-splitting",
"numPaths": 761,
"numPathLocations": 29790
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 28165
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 28165
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 28165
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 35765
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 35765
},
{
"query_id": "java/http-response-splitting",
"numPaths": 760,
"numPathLocations": 28165
},
{
"query_id": "java/zipslip",
"numPaths": 2,
"numPathLocations": 5
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26797
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26797
},
{
"query_id": "java/path-injection",
"numPaths": 1528,
"numPathLocations": 69045
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33754
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32902
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33754
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27749
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27749
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 30992
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 31312
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27133
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27749
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27749
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 30992
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 31312
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32076
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32990
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32076
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 22187
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 22187
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 22187
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 21935
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 29659
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 29425
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27133
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26942
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26942
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26942
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26942
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26751
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 21935
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27749
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27749
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33563
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32138
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33563
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32138
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 30228
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 30228
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 27554
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 32226
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 25266
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 30698
},
{
"query_id": "java/path-injection",
"numPaths": 766,
"numPathLocations": 31764
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33666
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 28707
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33503
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 28381
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26089
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26147
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 33093
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 35003
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 36531
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 36149
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 34621
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 34621
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 36149
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 35385
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 34621
},
{
"query_id": "java/path-injection",
"numPaths": 2,
"numPathLocations": 38
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26853
},
{
"query_id": "java/path-injection",
"numPaths": 764,
"numPathLocations": 26797
},
{
"query_id": "java/cleartext-storage-in-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/cleartext-storage-in-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/cleartext-storage-in-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/cleartext-storage-in-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/cleartext-storage-in-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/cleartext-storage-in-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/xss",
"numPaths": 764,
"numPathLocations": 26794
},
{
"query_id": "java/xss",
"numPaths": 764,
"numPathLocations": 26794
},
{
"query_id": "java/xss",
"numPaths": 764,
"numPathLocations": 37913
},
{
"query_id": "java/xss",
"numPaths": 764,
"numPathLocations": 37201
},
{
"query_id": "java/xss",
"numPaths": 764,
"numPathLocations": 32902
},
{
"query_id": "java/xss",
"numPaths": 2,
"numPathLocations": 47
},
{
"query_id": "java/xss",
"numPaths": 1,
"numPathLocations": 9
},
{
"query_id": "java/xss",
"numPaths": 2,
"numPathLocations": 35
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 38569
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 36922
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 43327
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 34360
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 48817
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 40948
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 49549
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 40948
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 43144
},
{
"query_id": "java/ssrf",
"numPaths": 732,
"numPathLocations": 38752
},
{
"query_id": "java/unsafe-deserialization",
"numPaths": 763,
"numPathLocations": 29889
},
{
"query_id": "java/insecure-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/insecure-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/insecure-cookie",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/xxe",
"numPaths": 1,
"numPathLocations": 4
},
{
"query_id": "java/xxe",
"numPaths": 764,
"numPathLocations": 28693
},
{
"query_id": "java/xxe",
"numPaths": 765,
"numPathLocations": 32207
},
{
"query_id": "java/xxe",
"numPaths": 764,
"numPathLocations": 31628
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 51423
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 50563
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 45819
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 45255
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 46007
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 45819
},
{
"query_id": "java/sql-injection",
"numPaths": 752,
"numPathLocations": 45443
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 764,
"numPathLocations": 32520
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 764,
"numPathLocations": 32520
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 765,
"numPathLocations": 27560
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 764,
"numPathLocations": 27554
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 764,
"numPathLocations": 29082
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 1,
"numPathLocations": 4
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 764,
"numPathLocations": 30757
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 764,
"numPathLocations": 27554
},
{
"query_id": "java/unvalidated-url-redirection",
"numPaths": 765,
"numPathLocations": 29951
},
{
"query_id": "java/unsafe-hostname-verification",
"numPaths": 1,
"numPathLocations": 3
},
{
"query_id": "java/unsafe-hostname-verification",
"numPaths": 1,
"numPathLocations": 2
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 8,
"numPathLocations": 454
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 8,
"numPathLocations": 454
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 1,
"numPathLocations": 2
},
{
"query_id": "java/weak-cryptographic-algorithm",
"numPaths": 1,
"numPathLocations": 3
},
{
"query_id": "java/tainted-format-string",
"numPaths": 935,
"numPathLocations": 28797
},
{
"query_id": "java/command-line-injection",
"numPaths": 764,
"numPathLocations": 35282
},
{
"query_id": "java/implicit-cast-in-compound-assignment",
"numPaths": 0,
"numPathLocations": 0
},
{
"query_id": "java/implicit-cast-in-compound-assignment",
"numPaths": 0,
"numPathLocations": 0
}
]
Hi @adityasharad,
Thank you for looking at it, I'm already running CodeQL only for js and it was helpful. All is now OK there.
As soon as I will get some time (Log4j did not help so far :smirk:) I'll follow your 3rd recommendation...
@aeisenberg @adityasharad @JacquesLeRoux thank you for the information provided here. On Keycloak we have the same issue https://github.com/keycloak/keycloak/actions/runs/1865950407. Reading the discussion here helped to identify the issue at least, which is the number of flow paths per alert. It would be nice to have the steps here documented in the troubleshooting guide, or provide an alternative to suppress those alerts.
@JacquesLeRoux your project is not the only one facing those issues, at the same time, it seems that we are the only ones with a large Java codebase and running CodeQL :)
Hi @abstractj,
Since then I did not work on that being deeply involved in OFBiz security. It would be great indeed to not have to handle the number of flow paths per alert upstream of CodeQL