paper-qa icon indicating copy to clipboard operation
paper-qa copied to clipboard

Retrying relevance score prompt if score was a float or response isn't valid JSON

Open jamesbraza opened this issue 1 year ago • 0 comments

Our LLM response --> Context logic is a bit fragile:

  • https://github.com/Future-House/paper-qa/issues/747 shows the LLM responding with a float relevance score.
  • https://github.com/Future-House/paper-qa/issues/736 shows the LLM failing to respond with valid JSON (though actually the issue seems to have been a token limitation).

What currently happens is this blows up Docs.aquery.

However, this should not be a critical failure because:

  1. It's not in the caller's (e.g. an agent) control, it's a failure internal to paper-qa.
  2. We can retry the summary prompt once or twice, saying something like:
    • "We failed to parse valid JSON according to exception {exc!r}. Please respond with JSON."
    • "We failed to extract a valid score according to exception {exc!r}. Please provide an integer score in 1-10."

jamesbraza avatar Dec 04 '24 18:12 jamesbraza