[Feature Request] Support sending UBI queries to a remote instance of OpenSearch

[jzonthemtn]

Is your feature request related to a problem? Please describe

There may be cases in which the UBI queries should not be saved to the local instance of OpenSearch. In these cases, it is required to store this information in a remote OpenSearch instance.

Describe the solution you'd like

The ability to configure UBI in OpenSearch to send events to a remote OpenSearch instead of always indexing the queries into the local ubi_queries index. Data Prepper looks to be a good choice to support this via its http_source. When a UBI query is received by OpenSearch, it is sent to Data Prepper's http endpoint where the Data Prepper pipeline can take care of indexing the query per the pipeline steps. This provides flexibility in how the event is stored since Data Prepper is not limited to just an OpenSearch sink.

In the event Data Prepper exposes an OpenSearch-compatible API in time, UBI can use that API via the client to send the event. (Described in https://github.com/opensearch-project/data-prepper/issues/248) If not, UBI can be updated later to use this method.

Additional context

Related to the UBI RFC https://github.com/opensearch-project/OpenSearch/issues/12084 and UBI initial code in https://github.com/opensearch-project/OpenSearch/issues/13545.

This issue can be assigned to myself.

[andrross]

[Triage - attendees 1 2 3 4] @jzonthemtn Thanks for filing.

[jzonthemtn]

Closing this RFC because the initial implementation is now available at https://github.com/opensearch-project/user-behavior-insights and new issues can be created there.