[BUG][Concurrent Search] Search request for top anomalies from AD plugin is failing

[sohami]

Describe the bug

AD plugin exposes a top anomalies api which internally create a painless script based composite aggregation query. When concurrent search is enabled on a cluster, then the search request sometimes fails with error.

Ref: Top Anomalies API: https://opensearch.org/docs/latest/observing-your-data/ad/api/#search-top-anomalies

Input to the API:

GET _plugins/_anomaly_detection/detectors/uBtS6HkBmDH36MzijITq/results/_topAnomalies
{
    "size": 10,
    "category_field": ["host", "service"],
    "order": "severity",
    "start_time_ms": 1622333056000,
    "end_time_ms": 1622937856000
}

Example Query which it creates:

"query": {
        "bool": {
            "filter": {
                "term": {
                    "detector_id": "uBtS6HkBmDH36MzijITq"
                }
            }
        }
    },
    "aggs": {
        "multi_buckets" : {
            "composite": {
                "sources": [
                    {
                        "host": {
                            "terms": {
                                "script": {
                                    "source": """
                                    String value = null;
                                    if (params == null || params._source == null || params._source.entity == null) {
                                        return "";
                                    }
                                    for (item in params._source.entity) {
                                        if (item['name'] == "host") {
                                            value = item['value'];
                                            break;
                                        }
                                    }
                                    return value;
                                    """,
                                    "lang": "painless"
                                }
                            }
                        }
                    },
                    {
                        "service": {
                            "terms": {
                                "script": {
                                    "source": """
                                    String value = null;
                                    if (params == null || params._source == null || params._source.entity == null) {
                                        return "";
                                    }
                                    for (item in params._source.entity) {
                                        if (item['name'] == "service") {
                                            value = item['value'];
                                            break;
                                        }
                                    }
                                    return value;
                                    """,
                                    "lang": "painless"
                                }
                            }
                        }
                    }
                ]
            },
            "aggregations": {
                "max": {
                    "max": {
                        "field": "anomaly_grade"
                    }
                },
                "multi_buckets_sort": {
                    "bucket_sort": {
                        "sort": [
                            { "max": { "order": "desc" } } 
                        ],
                        "size": 10                                
                    }
                }
            }
        }
    }

Related component

Search:Query Capabilities

To Reproduce

1. Create an index with AD plugin result index mapping. Ref here
2. Ingest the dummy data
3. Run the query shared in the description above.

Test failing in the plugin has sample of the index mapping for entity field and logic to create the data which can be used to reproduce. Ref here

Expected behavior

The query should be successful

Additional Details

Plugins Anomaly Detection

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context Add any other context about the problem here.