ml-commons icon indicating copy to clipboard operation
ml-commons copied to clipboard

[BUG] Incorrect parseLLMOutput caused by unexpected format of action_input in LLM response

Open qianheng-aws opened this issue 1 year ago • 1 comments

What is the bug? If LLM returns an invalid json response with action_input, it cannot extract the action input correctly.

After adding my new tool whose description is as below, its action_input in LLM response cannot be extracted correctly sometimes.

GetCreateMonitorParametersTool: 
This is a tool that creates alerts by AI assistant. It takes some optional arguments named
name which defines the name of the monitor, 
index which defines the input index of the monitor, 
timeField which defines the time field of the above input index, default is `timestamp`,
dataFilters which defines the filters, you should use the dataFilters directly without any change in the context if it exits,
bucketValue, a numeric value defines the time range for the last, default is 1,
bucketUnitOfTime defines the time unit of the above time range, should be m, h or d. default is h,
triggerDefinitions which defines triggers for this monitor, required if use wants to create alert while no need if creating monitor(it should be constructed as a list of json like {name, severity, thresholdEnum(should be ABOVE, BELOW or EXACTLY) and thresholdValue(default is 1000)}

Its action_input could be like below sometimes:

"action_input": `{
    "name": "Flight cancellation alert",
    "index": "opensearch_dashboards_sample_data_flights", 
    "timeField": "timestamp",
    "dataFilters": [{"fieldName": [{"label": "Cancelled", "type": "text"}], "operator": "is", "fieldValue": true}],
    "bucketValue": 1,
    "bucketUnitOfTime": "h",
    "triggerDefinitions": [{
      "name": "Flight cancellation count", 
      "severity": "CRITICAL",
      "thresholdEnum": "ABOVE",
      "thresholdValue": 10
    }]
  }`

However, extractActionInput only match pattern

"action_input"\s*:\s*"((?:[^\"]|\")*)"

which only match " instead of ``` and induced to extract nothing but NULL from the response.

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

What is the expected behavior? The action_input should be a valid json string and could be extracted correctly from the response

What is your host/environment?

  • OS: [e.g. iOS]
  • Version [e.g. 22]
  • Plugins

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

qianheng-aws avatar Jun 17 '24 07:06 qianheng-aws