Implement dynamic community selection for global search

[bryanlimy]

Description

Implement dynamic community selection to improve global search result.

Currently, we select the community reports by specifying a specific community level, prior to the map-reduce operation, while fast, this operation could eliminate useful community reports (i.e. at lower levels) and/or include reports that are not informative to the query.

Here, we take advantage of the hierachical structure of the indexed dataset. We first ask the LLM to rate how relevant each level 0 community is with respect to the user query, we then traverse down the child node(s) if the current community report is deemed relevant. Ultimately reducing the number of community reports that get passed to the map-reduce operation in global search.

The updated example_notebooks/global_search.ipynb show an example of such reduction in community reports with the same user query.

Related Issues

[Reference any related issues or tasks that this pull request addresses.]

Proposed Changes

[List the specific changes made in this pull request.]

Checklist

• [x] I have tested these changes locally.
• [x] I have reviewed the code changes.
• [x] I have updated the documentation (if necessary).
• [ ] I have added appropriate unit tests (if applicable).

Additional Notes

[Add any additional notes or context that may be helpful for the reviewer(s).]