Add experimental flags to save and load the rule index between runs

What type of PR is this?

Feature

What package or component does this PR mostly affect?

resolve/index.go

What does this PR do? Why is it needed?

This PR implements experimental flags to save and load the Gazelle rule index between runs. When Gazelle is given an index file to load, it skips traversing the entire repo, and instead uses the cached data in the provided index.

To make this work, a few things needed to change:

• The index is now structured with (closer to) one entry per indexed rule, and doesn't retain references to the File or Rule object that created them. For this to work, resolution of embeds is performed lazily at index query time. This also has the effect of removing the need to explicitly finalize the index.
• buildUpdateRelMap from walk.go has been refactored into a helper object, so that it can also be used to filter the index as it is loaded.
• index.go now has a unit test.

Which issues(s) does this PR fix?

Fixes #1181

Other notes for review

• ~Removing the .Finish() method from the RuleIndex is technically a breaking change for plugin authors, although a pretty minor one that will probably only affect test code. We could leave it in and make it a no-op if this is a concern.~
• index.go cannot depend on walk.go due to forming an import cycle, so the exclude list is passed as a function. This seems to work fine.

@fmeum I finally got around to resurrecting this quite-old PR, which adds experimental support for saving and loading the rule index. This is intended to serve as a building block for other tooling (either company-specific or in an improved version of autogazelle) that understands what files have changed and executes Gazelle on only those directories.

Solving that second problem is tricky especially in a fully general and provably correct sense, so I think it makes sense to leave it out of Gazelle for now.

I left the new changes in a separate commit for easy reviewing, since @jayconrod looked at an earlier draft of this pretty thoroughly a year ago.

FWIW, there was an excellent talk on BUILD file generation at Google at Bazelcon 2023 https://youtu.be/IXimf4DCAoY?t=8020. Might be worth a watch.

@jayconrod Would you be available to give this a final look? I'm not too familiar with this part of the code, but can look into it if you are busy.

I don't know exactly what the UX should look like. There's a good chance you'd want CI to generate the index files, triggered by CI, and Gazelle might need to fetch those.

We have a "incremental index" mode in Uber: when users run "gazelle ", Gazelle updates (with help of a global index file) and indexes all build targets in and its subdirectories. All the entries in the global index file under is also updated. So both index and build files can be incrementally updated. We never need to run Gazelle from root to populate the index. In fact, that's not possible in Git sparse checkout.

Our "index file" is actually a long list of gazelle:resolve at the root BUILD.bazel file, which is not ideal, but I hope this PR would preserve the "incremental index" feature.