Flaky Test RestMLGuardrailsIT

[mingshl]

What is the bug? RestMLGuardrailsIT.testPredictRemoteModelSuccess failed in https://github.com/opensearch-project/ml-commons/pull/2380

How can one reproduce the bug?

    REPRODUCE WITH: gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLGuardrailsIT.testPredictRemoteModelSuccess" -Dtests.seed=3BAA4A652C7FFA21 -Dtests.security.manager=false -Dtests.locale=cs-CZ -Dtests.timezone=America/Buenos_Aires -Druntime.java=21



Suite: Test class org.opensearch.ml.rest.RestMLGuardrailsIT
org.opensearch.ml.rest.RestMLGuardrailsIT > testPredictRemoteModelSuccess FAILED
  2> REPRODUCE WITH: gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestMLGuardrailsIT.testPredictRemoteModelSuccess" -Dtests.seed=3BAA4A652C7FFA21 -Dtests.security.manager=false -Dtests.locale=cs-CZ -Dtests.timezone=America/Buenos_Aires -Druntime.java=21
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:50733/], URI [/_plugins/_ml/models/hKDXMI8BN7KL5ERvDCOU/_predict], status line [HTTP/1.1 401 Unauthorized]
  2> org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:50733/], URI [/_plugins/_ml/models/hKDXMI8BN7KL5ERvDCOU/_predict], status line [HTTP/1.1 401 Unauthorized]
    {"error":{"root_cause":[{"type":"status_exception","reason":"Error from remote service: {\n    \"error\": {\n        \"message\": \"You didn't provide an API key. You need to provide your API key in an Authorization header using *** (i.e. Authorization: *** or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from [https://platform.openai.com/account/api-keys.\](https://platform.openai.com/account/api-keys./)",\n        \"type\": \"invalid_request_error\",\n        \"param\": null,\n        \"code\": null\n    }\n}\n"}],"type":"status_exception","reason":"Error from remote service: {\n    \"error\": {\n        \"message\": \"You didn't provide an API key. You need to provide your API key in an Authorization header using *** (i.e. Authorization: *** or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from [https://platform.openai.com/account/api-keys.\](https://platform.openai.com/account/api-keys./)",\n        \"type\": \"invalid_request_error\",\n        \"param\": null,\n        \"code\": null\n    }\n}\n"},"status":401}
    {"error":{"root_cause":[{"type":"status_exception","reason":"Error from remote service: {\n    \"error\": {\n        \"message\": \"You didn't provide an API key. You need to provide your API key in an Authorization header using *** (i.e. Authorization: *** or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from [https://platform.openai.com/account/api-keys.\](https://platform.openai.com/account/api-keys./)",\n        \"type\": \"invalid_request_error\",\n        \"param\": null,\n        \"code\": null\n    }\n}\n"}],"type":"status_exception","reason":"Error from remote service: {\n    \"error\": {\n        \"message\": \"You didn't provide an API key. You need to provide your API key in an Authorization header using *** (i.e. Authorization: *** or as the password field (with blank username) if you're accessing the API from your browser and are prompted for a username and password. You can obtain an API key from [https://platform.openai.com/account/api-keys.\](https://platform.openai.com/account/api-keys./)",\n        \"type\": \"invalid_request_error\",\n        \"param\": null,\n        \"code\": null\n    }\n}\n"},"status":401}
        at __randomizedtesting.SeedInfo.seed([3BAA4A652C7FFA21:AB434FFAEBBBB9D1]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:181)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:154)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:143)
        at app//org.opensearch.ml.rest.RestMLGuardrailsIT.predictRemoteModel(RestMLGuardrailsIT.java:230)
        at app//org.opensearch.ml.rest.RestMLGuardrailsIT.testPredictRemoteModelSuccess(RestMLGuardrailsIT.java:97)

[dhrubo-os]

This is a Flaky test.