[Auto Suggest] PPL & SQL Value Suggestion

[paulstn]

Description

SQL:

https://github.com/user-attachments/assets/54518f20-f349-4bc0-840e-9a38767b1aba

PPL:

https://github.com/user-attachments/assets/71c41b93-e199-48f8-acf2-7ba5aed16f73

src/plugins/data/public/antlr/*

For PPL & SQL, uses the SQL query

SELECT <column> FROM <table> GROUP BY <column> ORDER BY COUNT(<column>) DESC LIMIT <limit>

to find the most popular column values to display within autocomplete whenever a value should be suggested.

src/plugins/data/server/ui_settings.ts

There were two UI settings added. query:enhancements:suggestValues is a boolean that determines if value suggestion will be used query:enhancements:suggestValuesLimit is a number that specifies how many values should be queried, defaulting to 200

src/plugins/data/public/autocomplete/autocomplete_service.ts

The ability to add a value suggestion provided was included in this pr, but isn't in use so far. Creating a value suggestion provider that would make the api call (with all of the same parameters that exist right now) could be done but would require a lot of complexity for what is essentially done in 10 lines today.

This PR also contains various fixes and quality of life updates within SQL & PPL suggestions.

Issues Resolved

Screenshot

Testing the changes

Field level security testing:

https://github.com/user-attachments/assets/77bae369-8a30-42a3-acbc-bbab07184ed3

Field level masking testing:

https://github.com/user-attachments/assets/a7bc7f93-be32-4c72-9ce9-8dd9a15179ff

Changelog

• feat: Autocomplete Value Suggestion

Check List

• [ ] All tests pass
  • [ ] yarn test:jest
  • [ ] yarn test:jest_integration
• [ ] New functionality includes testing.
• [ ] New functionality has been documented.
• [ ] Update CHANGELOG.md
• [ ] Commits are signed per the DCO using --signoff

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 75.44910% with 41 lines in your changes missing coverage. Please review.

Project coverage is 61.61%. Comparing base (82689bb) to head (26782e2). Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
...ntlr/opensearch_sql/opensearch_sql_autocomplete.ts	68.62%	12 Missing and 4 partials :warning:
...ntlr/opensearch_ppl/opensearch_ppl_autocomplete.ts	63.88%	7 Missing and 6 partials :warning:
...ata/public/antlr/opensearch_sql/code_completion.ts	78.78%	3 Missing and 4 partials :warning:
...ugins/data/public/ui/query_editor/query_editor.tsx	0.00%	3 Missing :warning:
...ata/public/antlr/opensearch_ppl/code_completion.ts	81.81%	1 Missing and 1 partial :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8275      +/-   ##
==========================================
+ Coverage   61.01%   61.61%   +0.59%     
==========================================
  Files        3812     3813       +1     
  Lines       91385    91610     +225     
  Branches    14438    14498      +60     
==========================================
+ Hits        55762    56441     +679     
+ Misses      32065    31587     -478     
- Partials     3558     3582      +24     

Flag	Coverage Δ
Linux_1	29.00% <5.48%> (-0.08%)	:arrow_down:
Linux_2	?
Linux_3	39.08% <75.44%> (+1.05%)	:arrow_up:
Linux_4	28.93% <5.48%> (-0.10%)	:arrow_down:
Windows_1	29.01% <5.48%> (-0.08%)	:arrow_down:
Windows_2	56.40% <ø> (+<0.01%)	:arrow_up:
Windows_3	39.08% <75.44%> (+1.06%)	:arrow_up:
Windows_4	28.93% <5.48%> (-0.10%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[github-actions[bot]]

❌ Empty Changelog Section

The Changelog section in your PR description is empty. Please add a valid changelog entry or entries. If you did add a changelog entry, check to make sure that it was not accidentally included inside the comment block in the Changelog section.

[mengweieric]

Just summarize some of the concerns for this PR that I feel we need to address:

Grammar Changes: Modifying the grammar directly is not considered an optimal approach, as it introduces risks of mismatches with removed or modified grammar rules. We should prioritize leveraging visitors, token streams, and logical code adjustments to achieve the intended behavior.

Test Coverage: There is insufficient test coverage to ensure the modified grammar behaves consistently with previous expectations. Additional tests should be added to confirm the correctness of the changes if we want to go ahead with grammar changes.

Value Suggestions: While value suggestions are not a typical feature for SQL/PPL languages, it is acceptable in this context. However, the current implementation still has gaps in providing a meaningful suggestion experience, such as suggesting masked values.

Unaddressed Comments: Some comments have not been addressed yet. Please ensure that all feedback is reviewed and handled accordingly.

[opensearch-trigger-bot[bot]]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-8275-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a7aeb76b70b79880ff807895df3bb5cc19eb1178
# Push it to GitHub
git push --set-upstream origin backport/backport-8275-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8275-to-2.x.