[BUG] Error "Async Query APIs are disabled" when trying to access prometheus

[shotor]

Describe the bug

I updated to 2.13.0. Now when I go to datasources -> prometheus I get the following error when trying to query:

Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them.

GET /_cluster/settings?include_defaults=true&flat_settings=true
=> 
    "plugins.query.executionengine.async_query.enabled": "true",
    "plugins.query.executionengine.spark.config": "",

For some reason it's an empty string. I don't see any documentation on how to configure this. Trying to set it to an empty object yields an error:

PUT /_cluster/settings
{
  "persistent": {
    "plugins.query.executionengine.spark.config": {}
  }
}
=> 
{
  "error": {
    "root_cause": [
      {
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: no settings to update;"
      }
    ],
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: no settings to update;"
  },
  "status": 400
}

Unfortunately I can't downgrade to 2.12.0 because it gives me a formatTooNew error.

Related component

Search:Query Capabilities

To Reproduce

1. Add prometheus datasource to dashboard
2. Click on it
3. See error

Expected behavior

It should allow me to query the datasource

Additional Details

Plugins

• Security
• Async query

Screenshots

Host/Environment (please complete the following information):

• OS: debian12
• Docker: 26.0.0
• Version 2.13.0

Additional context

Full opensearch api log:

[2024-04-03T17:13:51,816][ERROR][o.o.s.s.r.RestAsyncQueryManagementAction] [9548ee319f74] Error happened during request handling
java.lang.IllegalArgumentException: Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them.
	at org.opensearch.sql.spark.client.EMRServerlessClientFactoryImpl.validateSparkExecutionEngineConfig(EMRServerlessClientFactoryImpl.java:52) ~[spark-2.13.0.0.jar:?]
	at org.opensearch.sql.spark.client.EMRServerlessClientFactoryImpl.getClient(EMRServerlessClientFactoryImpl.java:36) ~[spark-2.13.0.0.jar:?]
	at org.opensearch.sql.spark.dispatcher.SparkQueryDispatcher.dispatch(SparkQueryDispatcher.java:62) ~[spark-2.13.0.0.jar:?]
	at org.opensearch.sql.spark.asyncquery.AsyncQueryExecutorServiceImpl.createAsyncQuery(AsyncQueryExecutorServiceImpl.java:43) ~[spark-2.13.0.0.jar:?]
	at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:67) [spark-2.13.0.0.jar:?]
	at org.opensearch.sql.spark.transport.TransportCreateAsyncQueryRequestAction.doExecute(TransportCreateAsyncQueryRequestAction.java:28) [spark-2.13.0.0.jar:?]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395) [opensearch-security-2.13.0.0.jar:2.13.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.13.0.0.jar:2.13.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) [opensearch-performance-analyzer-2.13.0.0.jar:2.13.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.13.0.jar:2.13.0]
	at org.opensearch.sql.spark.rest.RestAsyncQueryManagementAction.lambda$executePostRequest$1(RestAsyncQueryManagementAction.java:126) [spark-2.13.0.0.jar:?]
	at org.opensearch.sql.datasources.utils.Scheduler.lambda$withCurrentContext$0(Scheduler.java:30) [datasources-2.13.0.0.jar:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.13.0.jar:2.13.0]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Dashboard logs:

Error in running direct query: StatusCodeError: Bad Request
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 400,
  displayName: 'BadRequest',
  path: '/_plugins/_async_query',

  query: {},
  body: '{\n' +
    '  "status": 400,\n' +
    '  "error": {\n' +
    '    "type": "IllegalArgumentException",\n' +
    '    "reason": "Invalid Request",\n' +
    '    "details": "Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them."\n' +
    '  }\n' +
    '}',
  statusCode: 400,
  response: '{\n' +
    '  "status": 400,\n' +
    '  "error": {\n' +
    '    "type": "IllegalArgumentException",\n' +
    '    "reason": "Invalid Request",\n' +
    '    "details": "Async Query APIs are disabled. Please configure plugins.query.executionengine.spark.config in cluster settings to enable them."\n' +
    '  }\n' +
    '}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8] @opensearch-project/admin Could you please transfer this SQL repo - it looks like they will be in the best position to look into this 2.13 impacting issue.

[HungryHowies]

I have the same issue, Upgrade to version 2.12.0 was when i first noticed this and then upgraded to version 2.13.0 still same issue. Using either API command or Prometheus Plugin Web UI I get a connection when querying Data I see the metrics but if I click on any one of them screen goes blank. I posted here

[AdaptiveStep]

have this problem too. Get the same "Async Query APIs are disabled" error. Its really hard to get opensearch to work with any kind of prometheus stuff..

[szydell]

In 2.14.0 i can connect to Thanos again. It does not work too well, but that's for another ticket. :)