EdgeChains icon indicating copy to clipboard operation
EdgeChains copied to clipboard
verified publisher icon arakoodev

eval/feedback system

Open sandys opened this issue 2 years ago • 1 comments

image

edgechains must have built in support to capture feedback for prompts/final answers (just like for token counts which we already do)

sandys avatar Sep 06 '23 05:09 sandys

  1. User feedback: thumbs up/down associated to a piece of data that I can trace back to a workflow run that used a specific version of a prompt/template
  2. Output metrics: certain thresholds that flag LLM-generated output (in my case it's related to ecommerce search; so if something turns up high on search results but low on click/conversion it could be a telling me that my prompt is generating bad data)

sandys avatar Sep 20 '23 09:09 sandys