llmine_core icon indicating copy to clipboard operation
llmine_core copied to clipboard

Add strong typed validation for LLM returned values

Open arpanpreneur opened this issue 1 year ago • 0 comments

Background

We have well defined jsonschema and labels for extractor prompts for json and labels return types respectively. But currently we don't have any validation after the LLM has responded to an extractor prompt.

New Behaviour

  1. Check for json type: If returned JSON text from the LLM does not conforms to the given jsonschema use separate LLM call to correct it or retry the prompt.
  2. Check for labels type: Returned string must be one of the many values in thelabels_config_json array.
  3. Text type needs no special validation and retry

Where to make such changes

There is a TODO comment in Line 77 in tasks.py in run_extracter_chain method.

arpanpreneur avatar Sep 19 '23 18:09 arpanpreneur