[Feature][MDS] TSVB support with Multi data source

[bandinib-amzn]

Problem Statement

As of today, the Time Series Visual Builder (TSVB) - a data visualization tool in Dashboards - does not support remote data connections. However, Dashboards can connect to any remotely located OpenSearch instance through connections created using the data source feature.

Background

Currently TSVB is tightly integrated with OpenSearch. TSVB directly queries OpenSearch to retrieve the time-series data needed for visualization. TSVB relies on index patterns defined in OpenSearch-Dashboards to identify the OpenSearch indices containing the relevant data.

There are two APIs /api/metrics/fields and /api/metrics/vis/data which are part of the Time Series Visual Builder (TSVB) plugin in OpenSearch-Dashboards. They are used for fetching field metadata and metric data for visualizations, respectively.

1. /api/metrics/fields: This API endpoint is responsible for providing information about the fields available in the indices that TSVB can query. It returns metadata such as the data type, format, and other properties of the fields. This information is crucial for configuring TSVB visualizations as it helps users understand the structure of the data they are working with.

2. /api/metrics/vis/data: This API endpoint is used to retrieve metric data for TSVB visualizations. It allows users to specify parameters such as the time range, interval, and other options to fetch the relevant metric data from OpenSearch. The data retrieved from this endpoint is then used to render the TSVB visualization in the OpenSearch-Dashboards.

These APIs are essential for the functionality of TSVB as they enable users to interact with OpenSearch and fetch the data needed for creating and displaying time-series visualizations in Dashboards. To enable TSVB to connect and retrieve data from a remote cluster, we need to add support for data sources within these two APIs.

Solution

1] Modify Index Pattern selector under Panel Options and Annotation:

The current input type of the Index pattern selector in TSVB is a text field. We need to extend the functionality of this text fields to support index pattern of format [Data Source]::[Index Pattern].

2] Modify /api/metrics/fields to accept dataSourceId as query parameter:

2.1 Add data source id as query parameter

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2c8d9d39028bf5fff38e4b0354765039aa2e7284/src/plugins/vis_type_timeseries/server/routes/fields.ts#L36-L43

2.2 And then pass to getFields

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2c8d9d39028bf5fff38e4b0354765039aa2e7284/src/plugins/vis_type_timeseries/server/routes/fields.ts#L46

2.3 Then inside getFields, based on dataSourceId, we need to IndexPatternsFetcher with correct client.

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2c8d9d39028bf5fff38e4b0354765039aa2e7284/src/plugins/vis_type_timeseries/server/lib/get_fields.ts#L58-L61

3] Modify /api/metrics/vis/data to accept dataSourceId as query parameter:

/api/metrics/vis/data uses data plugin for search and query OpenSearch. Data plugin already have capability to accept dataSourceID and decide client based on context and request.

3.1 Add data source id as query parameter https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2c8d9d39028bf5fff38e4b0354765039aa2e7284/src/plugins/vis_type_timeseries/server/routes/vis.ts#L45-L51

3.2 Then pass dataSourceId to search.

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2c8d9d39028bf5fff38e4b0354765039aa2e7284/src/plugins/vis_type_timeseries/server/lib/search_strategies/strategies/abstract_search_strategy.ts#L74-L82

Alternate Solution

For 1st point in proposed solution, instead of extending the functionality of this text fields to support index pattern of format [Data Source]::[Index Pattern], use same component which is being used in Discover.