[BUG] Observability-Metrics fails to render page with Prometheus data source metric selected

[lyradc]

What is the bug? With a Prometheus data source added the Observability -> Metrics will show the available metrics from this data source. However when a metric is selected the page becomes mostly empty and errors are thrown in the browser console.

How can one reproduce the bug? Steps to reproduce the behavior:

1. Create a Prometheus data source (below ran from Dev Tools)

POST _plugins/_query/_datasources
{
  "name": "prometheus",
  "connector": "prometheus",
  "properties": {
    "prometheus.uri": "https://prometheus.url.com/",
    "prometheus.auth.type": "basicauth",
    "prometheus.auth.username": "username",
    "prometheus.auth.password": "password"
  },
  "allowedRoles": []
}

2. Navigate to Observability -> Metrics

3. Wait for the list of available metrics to populate

4. Click on any metric, a moment later the page clears

When the page clears the following errors present in the browser console

What is the expected behavior? When a metric is selected from Observability -> Metrics I expect to see a summary of the metric similar to the screenshots in https://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/

What is your host/environment?

• OS: Docker deployed on CentOS7
• Version 2.12.0
• Browser: Chrome version 122.0.6261.95 (Official Build) (64-bit)
• Relevant Plugins: opensearch-sql version 2.12.0.0 opensearch-observability version 2.12.0.0

[lyradc]

Possibly a factor, our Prometheus instance contains >1k metrics across ~600 servers.

Afterward, following removing all datasources previously defined, we then found Kibana was becoming increasingly unresponsive. I was unable to capture details on this behavior as it occurred in our much larger production environment and was not repeatable in smaller dev env. Disabling data sources in opensearch_dashboards.yml recovered Kibana's performance.

[penghuo]

@vamsi-amazon please take a look.

[HungryHowies]

I'm having the same issue on OS/OSD v2.12.0 on Ubuntu 22.0.4

• I connected to Prometheus in the Data source section

• Checked my datasource index set…

• On the left Pane Observability → Metrics I see this

• If I click on any one of those metrics or I click on Query Data link I see this…

Once that happens I never can see metrics list again. I have tried Firefox and almost seen a visualization started and it went blank.

[akshatmisra198808]

Same has been happening with my environment. I would like to know if I can revert back to older version, will that help? Also, I want to validate if the data source(via api or dev tool) has the data saved(prometheus). I want to use grafana opensearch plugin (https://grafana.com/grafana/plugins/grafana-opensearch-datasource/) to visualize the data for my June release. I want to get the data from opensearch.

[nateynateynate]

I can't even make it this far. I've got prometheus and the node_exporter running, but setting up the data source then viewing data sources fails.

Following the instructions here will get you to this point: https://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/

[SergioFernandezCordero]

Hi I'm facing the same issue when adding prometheus datastore from the OpenSearch Management Portal. I have a local instance where I have easy to test, if that helps.

[HungryHowies]

I can't even make it this far. I've got prometheus and the node_exporter running, but setting up the data source then viewing data sources fails.

Following the instructions here will get you to this point: https://opensearch.org/docs/latest/observing-your-data/prometheusmetrics/

Hey, So on slack I talk to someone and even version 2.13 failed. I think from what I understand this should be fixed on ver 2.14

[vamsi-amazon]

@HungryHowies can you point me to the slack thread. Sorry for the delayed response, I will try to get this fixed asap. @nateynateynate , @HungryHowies

I'm having the same issue on OS/OSD v2.12.0 on Ubuntu 22.0.4

• I connected to Prometheus in the Data source section

• Checked my datasource index set…

• On the left Pane Observability → Metrics I see this

• If I click on any one of those metrics or I click on Query Data link I see this…

Once that happens I never can see metrics list again. I have tried Firefox and almost seen a visualization started and it went blank.

@HungryHowies Sorry for the delayed response. do we have more details(console logs/har file) to understan the issue better?

[HungryHowies]

@HungryHowies can you point me to the slack thread. Sorry for the delayed response, I will try to get this fixed asap. @nateynateynate , @HungryHowies

I'm having the same issue on OS/OSD v2.12.0 on Ubuntu 22.0.4

• I connected to Prometheus in the Data source section

• Checked my datasource index set…

• On the left Pane Observability → Metrics I see this

• If I click on any one of those metrics or I click on Query Data link I see this…

Once that happens I never can see metrics list again. I have tried Firefox and almost seen a visualization started and it went blank.

@HungryHowies Sorry for the delayed response. do we have more details(console logs/har file) to understan the issue better? Slack Convo https://opensearch.slack.com/archives/C01QENNTGUD/p1711760080485209

[AdaptiveStep]

Does the datasource prometheus in opensearch work for anyone?

I can't make it work, it keeps saying:

{
  "status": 400,
  "error": {
    "type": "IllegalArgumentException",
    "reason": "Invalid Request",
    "details": "Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them."
  }
}

[nateynateynate]

I can't make it work, it keeps saying:

I was also unable. I think our only current course of action is to wait until 2.14 releases. Here's the current release schedule: https://opensearch.org/releases.html

[vamsi-amazon]

@RyanL1997 did this get fixed in 2.14?

[HungryHowies]

I just install OS 2.14 and it started and shows my metrics names but no data

[RyanL1997]

https://github.com/opensearch-project/sql/issues/2541#issuecomment-2078196267

This surfaced up again in current main and 2.14, which is introduced by: https://github.com/opensearch-project/dashboards-observability/pull/1739. We just did the fixes in main:

• https://github.com/opensearch-project/dashboards-observability/pull/1830

[SergioFernandezCordero]

Upgrading to 2.14 solved the problem for me. Thank you!