ml-commons
ml-commons copied to clipboard
[ML Inference Search Processors] Always return list when using dollar symbol in input_maps
Description
fixing https://github.com/opensearch-project/ml-commons/issues/2974
by adding a helpful method getMappedInputFromObject when reading search query/ search response _source document,
* Retrieves the mapped input value from the object based on the provided field name.
* when the field name start with a dollar symbol, which starts all path expressions in JSONPATH notation,
* default to always return lists
* otherwise, return the original format of elements retrieving from the fieldName
For example,
in search processors:
for the same item index, if configuring input_map as {"input": "$.item.text"},
the model input will be in list representation.
{
"input": ["red shoes"]
}
for the same item index, if configuring input_map as {"input": "item.text"},
the model input will be in list representation.
{
"input": "red shoes"
}
Check List
- [ ] New functionality includes testing.
- [ ] New functionality has been documented.
- [ ] API changes companion pull request created.
- [ ] Commits are signed per the DCO using
--signoff. - [ ] Public documentation issue/PR created.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.
rerunning flaky tests
RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent" -Dtests.seed=FF62341AF71E5511 -Dtests.security.manager=false -Dtests.locale=cv-Cyrl-RU -Dtests.timezone=America/Thunder_Bay -Druntime.java=21
RestConnectorToolIT > testConnectorToolInFlowAgent FAILED
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent" -Dtests.seed=FF62341AF71E5511 -Dtests.security.manager=false -Dtests.locale=cv-Cyrl-RU -Dtests.timezone=America/Thunder_Bay -Druntime.java=21
org.opensearch.client.ResponseException: method [POST], host [http://[::1]:42541], URI [/_plugins/_ml/agents/2WmEIJIBugkMbA_Nmc3O/_execute], status line [HTTP/1.1 500 Internal Server Error]
{"status":500,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error communicating with remote model: bedrock-runtime.us-west-2.amazonaws.com: Name or service not known"}}
at __randomizedtesting.SeedInfo.seed([FF62341AF71E5511:DAC9050B68D574D9]:0)
at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
at app//org.opensearch.ml.rest.RestBaseAgentToolsIT.executeAgent(RestBaseAgentToolsIT.java:84)
at app//org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent(RestConnectorToolIT.java:140)
RestCon
after considering standardization across ml commons repo, taking another fix https://github.com/opensearch-project/ml-commons/pull/2985 closing this PR