sql icon indicating copy to clipboard operation
sql copied to clipboard

[BUG] Circuit breaker getting triggered when multiple PPL queries are fired in parallel.

Open RyanL1997 opened this issue 1 month ago • 5 comments

Description

This issue transfers a experiment @paulstn and I did in OSD's new discover experience. When loading into the discover page at OSD, it fires multiple queries against each index field to calculate the presentation percentage.

  • Similar issue https://github.com/opensearch-project/sql/issues/4584

Example queries:

# query 1
source = demo-ai-logs-otel-v1* 
  | WHERE `@timestamp` >= '2025-07-29 22:53:20.604' AND `@timestamp` <= '2025-10-27 22:53:20.604' 
  | where isnotnull(`attributes.State`) 
  | stats count() as field_count, distinct_count(`attributes.State`) as distinct_count

# query 2
source = demo-ai-logs-otel-v1* 
  | WHERE `@timestamp` >= '2025-07-29 22:53:20.604' AND `@timestamp` <= '2025-10-27 22:53:20.604' 
  | where isnotnull(`attributes.address`) | stats count() as field_count, distinct_count(`attributes.address`) as distinct_count
...

Example errors:

...
{
  "text": "{\"statusCode\":400,\"error\":\"Bad Request\",\"message\":\"{\\n  \\\"error\\\": {\\n    \\\"reason\\\": \\\"Error occurred in OpenSearch   engine: all shards failed\\\",\\n    
  \\\"details\\\": \\\"Shard[0]: [demo-ai-logs-otel-v1-00001/oIkyldJtSHm6Cwgnj-aqxA] QueryShardException[failed to create query: 
  Failed to compile inline script   [rO0ABXNyADRvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTCQyPc501CEBPWwCAAVMAA12YWwkYXJn
  ...
  ...
  dGltZS5TZXKVXYS6GyJIsgwAAHhwdwYHAANVVEN4c3EAfgBNdw0CAAAAAGj/9+AtYkYHeH5xAH4AG3QAB0JPT0xFQU4=] using lang
  [opensearch_query_expression]]; 
  nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, 
  or scripts with parameters instead; this limit can be changed by the [script.context.filter.max_compilations_rate] setting];\\\\n\\\\n
  For more details, please send request for Json format to see the raw response from OpenSearch engine.\\\",\\n    
  \\\"type\\\": \\\"SearchPhaseExecutionException\\\"\\n  },\\n  \\\"status\\\": 400\\n}\"}"
  },
  "redirectURL": "",
  "headersSize": 800,
  "bodySize": 3641
},
...

Env

  • OS: 3.3

Screenshots

Image

Potential Work Around

Instead of having multiple queries fired front frontend, do a single query.

Exit Criteria:

As SQL plugin, here are couple action items:

  • [ ] Investigate the current default limitation of script.context.filter.max_compilations_rate
  • [ ] Implement the fix/enhancement to address the above issue

RyanL1997 avatar Nov 10 '25 19:11 RyanL1997