ml-commons icon indicating copy to clipboard operation
ml-commons copied to clipboard

[ML Inference Search Processors] Always return list when using dollar symbol in input_maps

Open mingshl opened this issue 1 year ago • 1 comments

Description

fixing https://github.com/opensearch-project/ml-commons/issues/2974

by adding a helpful method getMappedInputFromObject when reading search query/ search response _source document,

 * Retrieves the mapped input value from the object based on the provided field name.
 * when the field name start with a dollar symbol, which starts all path expressions in JSONPATH notation,
 * default to always return lists
 * otherwise, return the original format of elements retrieving from the fieldName

For example,

in search processors:

for the same item index, if configuring input_map as {"input": "$.item.text"},

the model input will be in list representation.

{
  "input": ["red shoes"]
}

for the same item index, if configuring input_map as {"input": "item.text"},

the model input will be in list representation.

{
  "input": "red shoes"
}

Check List

  • [ ] New functionality includes testing.
  • [ ] New functionality has been documented.
  • [ ] API changes companion pull request created.
  • [ ] Commits are signed per the DCO using --signoff.
  • [ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

mingshl avatar Sep 23 '24 19:09 mingshl

rerunning flaky tests

RestConnectorToolIT > testConnectorToolInFlowAgent STANDARD_ERROR
    REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent" -Dtests.seed=FF62341AF71E5511 -Dtests.security.manager=false -Dtests.locale=cv-Cyrl-RU -Dtests.timezone=America/Thunder_Bay -Druntime.java=21


RestConnectorToolIT > testConnectorToolInFlowAgent FAILED
REPRODUCE WITH: ./gradlew ':opensearch-ml-plugin:integTest' --tests "org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent" -Dtests.seed=FF62341AF71E5511 -Dtests.security.manager=false -Dtests.locale=cv-Cyrl-RU -Dtests.timezone=America/Thunder_Bay -Druntime.java=21
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:42541], URI [/_plugins/_ml/agents/2WmEIJIBugkMbA_Nmc3O/_execute], status line [HTTP/1.1 500 Internal Server Error]
    {"status":500,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error communicating with remote model: bedrock-runtime.us-west-2.amazonaws.com: Name or service not known"}}
        at __randomizedtesting.SeedInfo.seed([FF62341AF71E5511:DAC9050B68D574D9]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:182)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:155)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:144)
        at app//org.opensearch.ml.rest.RestBaseAgentToolsIT.executeAgent(RestBaseAgentToolsIT.java:84)
        at app//org.opensearch.ml.rest.RestConnectorToolIT.testConnectorToolInFlowAgent(RestConnectorToolIT.java:140)

RestCon

mingshl avatar Sep 23 '24 20:09 mingshl

after considering standardization across ml commons repo, taking another fix https://github.com/opensearch-project/ml-commons/pull/2985 closing this PR

mingshl avatar Oct 01 '24 22:10 mingshl