Feature: add configuration to enable analyzers for querying OpenSearch workspaces

[massi-ang]

The chatbot currently supports only exact keyword search when using hybrid search configuration for the workspace. While this setting works fine for some type of documents, it might not be performing well with textual searches in natural language, since stopwords will also result in matches and stemming is not applied (https://opensearch.org/docs/latest/query-dsl/full-text/match/#analyzer)

The proposal is to add a flag to the workspace to enable analyzers, in which case the query would be performed with an analyzer matching the language in which the query is performed, if the language among those configured on the workspace or the default language for the workspace.

A default workspace language setting should also be implemented, or a reasonable default used.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[ystoneman]

Could you please share a bit more to help us better understand what you're looking for, @massi-ang?

1. An example showing the benefits of a default workspace language and automatic query language detection, including:

   • Sample queries
   • Current output
   • Desired behavior with the proposed changes

2. Details on when the default language should be used and if it should fallback for unsupported languages.

3. Any recommended libraries or APIs for language detection that can handle ambiguous cases.

4. An explanation of how the workspace's "languages" setting should interact with these features and if it should limit query languages.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.