sql
sql copied to clipboard
[BUG] Circuit breaker getting triggered when multiple PPL queries are fired in parallel.
Description
This issue transfers a experiment @paulstn and I did in OSD's new discover experience. When loading into the discover page at OSD, it fires multiple queries against each index field to calculate the presentation percentage.
- Similar issue https://github.com/opensearch-project/sql/issues/4584
Example queries:
# query 1
source = demo-ai-logs-otel-v1*
| WHERE `@timestamp` >= '2025-07-29 22:53:20.604' AND `@timestamp` <= '2025-10-27 22:53:20.604'
| where isnotnull(`attributes.State`)
| stats count() as field_count, distinct_count(`attributes.State`) as distinct_count
# query 2
source = demo-ai-logs-otel-v1*
| WHERE `@timestamp` >= '2025-07-29 22:53:20.604' AND `@timestamp` <= '2025-10-27 22:53:20.604'
| where isnotnull(`attributes.address`) | stats count() as field_count, distinct_count(`attributes.address`) as distinct_count
...
Example errors:
...
{
"text": "{\"statusCode\":400,\"error\":\"Bad Request\",\"message\":\"{\\n \\\"error\\\": {\\n \\\"reason\\\": \\\"Error occurred in OpenSearch engine: all shards failed\\\",\\n
\\\"details\\\": \\\"Shard[0]: [demo-ai-logs-otel-v1-00001/oIkyldJtSHm6Cwgnj-aqxA] QueryShardException[failed to create query:
Failed to compile inline script [rO0ABXNyADRvcmcub3BlbnNlYXJjaC5zcWwuZXhwcmVzc2lvbi5mdW5jdGlvbi5GdW5jdGlvbkRTTCQyPc501CEBPWwCAAVMAA12YWwkYXJn
...
...
dGltZS5TZXKVXYS6GyJIsgwAAHhwdwYHAANVVEN4c3EAfgBNdw0CAAAAAGj/9+AtYkYHeH5xAH4AG3QAB0JPT0xFQU4=] using lang
[opensearch_query_expression]];
nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed,
or scripts with parameters instead; this limit can be changed by the [script.context.filter.max_compilations_rate] setting];\\\\n\\\\n
For more details, please send request for Json format to see the raw response from OpenSearch engine.\\\",\\n
\\\"type\\\": \\\"SearchPhaseExecutionException\\\"\\n },\\n \\\"status\\\": 400\\n}\"}"
},
"redirectURL": "",
"headersSize": 800,
"bodySize": 3641
},
...
Env
- OS: 3.3
Screenshots
Potential Work Around
Instead of having multiple queries fired front frontend, do a single query.
Exit Criteria:
As SQL plugin, here are couple action items:
- [ ] Investigate the current default limitation of
script.context.filter.max_compilations_rate - [ ] Implement the fix/enhancement to address the above issue