codeql
codeql copied to clipboard
github
QL: Untangle the various parsers
(WIP -- there's some weirdness surrounding QLpacks that is causing one of the tests to fail, see below.)
Splits the Frankengrammar apart into three smaller (and hopefully nicer) pieces.
I recommend reviewing commit-by-commit, as the changes to TreeSitter.qll are somewhat incomprehensible otherwise.
As mentioned above, currently one of the tests is failing:
With the latest commit, QLpack parsing is still broken, but in a different way. Now it causes the extractor to panic when running the callgraph test. Here's the output (with RUST_BACKTRACE=1) when this happens:
[2022-07-22 15:49:47] [build-err] Running command in /workspaces/codeql/ql/ql/test/callgraph: [/workspaces/codeql/ql/extractor-pack/tools/index-files.sh, /workspaces/codeql/ql/ql/test/callgraph/callgraph.testproj/working/files-to-index7746530681239887963.list]
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', extractor/src/extractor.rs:330:14
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` valuestack backtrace:
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] ', extractor/src/extractor.rs:330:14
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', extractor/src/extractor.rs:330:14
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stdout] ERROR /workspaces/codeql/ql/ql/test/callgraph/packs/other/qlpack.yml:1: parse error
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stdout] ERROR /workspaces/codeql/ql/ql/test/callgraph/packs/lib/qlpack.yml:1: parse error
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stdout] ERROR /workspaces/codeql/ql/ql/test/callgraph/packs/src/qlpack.yml:1: parse error
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 0: rust_begin_unwind
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/panicking.rs:584:5
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 1: core::panicking::panic_fmt
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:142:14
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 2: core::panicking::panic
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:48:5
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 3: ql_extractor::extractor::extract
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 5: rayon::iter::plumbing::Folder::consume_iter
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 6: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 7: rayon_core::registry::in_worker
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 8: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 9: rayon_core::registry::in_worker
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 10: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 11: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [ERROR] Spawned process exited abnormally (code 101; tried to run: [/workspaces/codeql/ql/extractor-pack/tools/index-files.sh, /workspaces/codeql/ql/ql/test/callgraph/callgraph.testproj/working/files-to-index7746530681239887963.list])
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 12: rayon_core::registry::WorkerThread::wait_until_cold
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 13: rayon_core::registry::ThreadBuilder::run
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] stack backtrace:
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 0: rust_begin_unwind
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/panicking.rs:584:5
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 1: core::panicking::panic_fmt
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:142:14
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 2: core::panicking::panic
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:48:5
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 3: ql_extractor::extractor::extract
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 5: rayon::iter::plumbing::Folder::consume_iter
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 6: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 7: rayon_core::registry::in_worker
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 8: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 9: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 10: rayon_core::registry::WorkerThread::wait_until_cold
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 11: rayon_core::registry::ThreadBuilder::run
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] stack backtrace:
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 0: rust_begin_unwind
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/panicking.rs:584:5
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 1: core::panicking::panic_fmt
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:142:14
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 2: core::panicking::panic
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/panicking.rs:48:5
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 3: ql_extractor::extractor::extract
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 4: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 5: rayon::iter::plumbing::Folder::consume_iter
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 6: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 7: rayon_core::registry::in_worker
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 8: rayon::iter::plumbing::bridge_producer_consumer::helper
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 9: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 10: rayon_core::registry::WorkerThread::wait_until_cold
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] 11: rayon_core::registry::ThreadBuilder::run
[2022-07-22 15:49:47] [build] [2022-07-22 15:49:47] [build-stderr] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2022-07-22 15:49:47] [build-err] A fatal error occurred: Exit status 101 from command: [/workspaces/codeql/ql/extractor-pack/tools/index-files.sh, /workspaces/codeql/ql/ql/test/callgraph/callgraph.testproj/working/files-to-index7746530681239887963.list]
[2022-07-22 15:49:47] [ERROR] Spawned process exited abnormally (code 2; tried to run: [/workspaces/codeql/ql/extractor-pack/tools/qltest.sh])
Could not extract a dataset in /workspaces/codeql/ql/ql/test/callgraph: Extraction command /workspaces/codeql/ql/extractor-pack/tools/qltest.sh failed with status 2.
Extraction command /workspaces/codeql/ql/extractor-pack/tools/qltest.sh failed with status 2.
[1/1] FAILED(EXTRACTION) /workspaces/codeql/ql/ql/test/callgraph/callgraph.ql
The problem is that you didn't regenerate the Rust & Node bindings in the branch containing the YAML parser: they are actually still calling the tree_sitter_ql C function.
https://github.com/tausbn/tree-sitter-ql/blob/c9d7d8de9fab75f39d5bc7bdf3a6e5151bb9b3bd/bindings/rust/lib.rs#L27-L29
Normally this would result in the linker complaining about an undefined symbol, but in this case we are also linking in the real tree_sitter_ql, so the linker is happy.
This means we're calling the QL parser to parse the YAML file, and then acting surprised when we get node types back that don't match the node-types.json file for the YAML parser.
The fix should be simple enough:
rm -r bindings
tree-sitter generate
You are also not using the same revision of tree-sitter-ql-yaml in the two Cargo.toml files (generator/Cargo.toml and extractor/Cargo.toml).
And it doesn't look like you've pinned the versions of tree-sitter-ql / tree-sitter-ql-dbscheme in the Cargo.toml files.
I've tried to fix it all of that, and I've also re-generated the bindings.
And I've changed the tree-sitter version constraint to ">= 0.20, < 0.21" in all the grammars.
Because some parts were compiled with 0.19 and some with 0.20.
~~And now we got plenty of parse errors, and thread panics.
I'll let you guys handle that. (Taus will be back from vacation next week).~~
Edit1: That was my mistake. I had to re-generate the bindings in the other grammars.
Things seem to work now (locally 🤞).
Edit2: And now we get some parse errors on the new uses of parametarized modules. I'm looking into that.
Edit3: And the grammar didn't support imports with module instantiations (example).
So I changed the parsing of import-statements to use the existing ModuleExpr rule.
That also allowed me to reuse the same logic as we already have for resolving ModuleExpr.
Edit4:
And I rebased the history on main to allow me to get a prettier history.
I'm almost done now.
~~There are 2 imports that we don't resolve uniquely (see the code-scanning results).
Both the modules that each import resolve to are valid targets of the imports, but CodeQL seems to have a priority mechanism that we don't emulate in QL-for-QL.
The CodeQL extension also produces a warning at those two imports, so I think it's OK for now.~~
I did a rebase in main and re-enabled the consistency checks on swift now that https://github.com/github/codeql/pull/10019 has been merged.