ml-commons
ml-commons copied to clipboard
[BUG] Incorrect parseLLMOutput caused by unexpected format of action_input in LLM response
What is the bug? If LLM returns an invalid json response with action_input, it cannot extract the action input correctly.
After adding my new tool whose description is as below, its action_input in LLM response cannot be extracted correctly sometimes.
GetCreateMonitorParametersTool:
This is a tool that creates alerts by AI assistant. It takes some optional arguments named
name which defines the name of the monitor,
index which defines the input index of the monitor,
timeField which defines the time field of the above input index, default is `timestamp`,
dataFilters which defines the filters, you should use the dataFilters directly without any change in the context if it exits,
bucketValue, a numeric value defines the time range for the last, default is 1,
bucketUnitOfTime defines the time unit of the above time range, should be m, h or d. default is h,
triggerDefinitions which defines triggers for this monitor, required if use wants to create alert while no need if creating monitor(it should be constructed as a list of json like {name, severity, thresholdEnum(should be ABOVE, BELOW or EXACTLY) and thresholdValue(default is 1000)}
Its action_input could be like below sometimes:
"action_input": `{
"name": "Flight cancellation alert",
"index": "opensearch_dashboards_sample_data_flights",
"timeField": "timestamp",
"dataFilters": [{"fieldName": [{"label": "Cancelled", "type": "text"}], "operator": "is", "fieldValue": true}],
"bucketValue": 1,
"bucketUnitOfTime": "h",
"triggerDefinitions": [{
"name": "Flight cancellation count",
"severity": "CRITICAL",
"thresholdEnum": "ABOVE",
"thresholdValue": 10
}]
}`
However, extractActionInput only match pattern
"action_input"\s*:\s*"((?:[^\"]|\")*)"
which only match " instead of ``` and induced to extract nothing but NULL from the response.
How can one reproduce the bug? Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
What is the expected behavior? The action_input should be a valid json string and could be extracted correctly from the response
What is your host/environment?
- OS: [e.g. iOS]
- Version [e.g. 22]
- Plugins
Do you have any screenshots? If applicable, add screenshots to help explain your problem.
Do you have any additional context? Add any other context about the problem.