codeql icon indicating copy to clipboard operation
codeql copied to clipboard

Ruby: Change how we pull in `shared/tree-sitter-extractor` dependency

Open criemen opened this issue 1 year ago • 1 comments

Previously, we pulled in the shared tree-sitter extractor via a git dependency in Cargo.toml to address a rules_rust limitation (no path dependencies outside of the cargo workspace)). This was a problem, as that means we're cloning github/codeql again for the build, which is quite slow.

I found another way that is faster, and still produces correct builds for both cargo`` and rules_rust`:

  • Cargo depends on a fake crate that has the same dependencies as the real crate (thanks to sync-files.py). Therefore, cargo pulls in the right dependencies into the lockfile, which bazel targets
  • For local builds, we override the path to that dependency in a cargo config, so we're pulling in the correct code
  • rules_rust only uses path dependencies for collecting transitive dependencies, it never pulls in the code from there. So far that, we manually provide a BUILD.bazel file for the shared extractor, and depend on that.

The git dependency was also a major source of slowdown for the upcoming bzlmod/rules_rust conversion, so we needed to get rid of it for that reason, too.

criemen avatar May 24 '24 09:05 criemen

Ah, one more note: If this weren't a public repository/windows support wouldn't be a concern, I'd have symlinked the files, or even the project directory, but that then creates problems for people checking out the repo on Windows that don't have symlinks enabled. All our developers need to have those enabled anyways for bazel to work, but external contributors probably haven't, so that's a non-starter.

criemen avatar May 24 '24 13:05 criemen