paper-qa icon indicating copy to clipboard operation
paper-qa copied to clipboard

Empty session.answer for exhaustive search without an answer

Open mskarlin opened this issue 1 year ago • 1 comments

The new complete tool allows the agent to call it quits after trying many search iterations. This is great but if it finds no relevant evidence, the GenerateAnswer tool won't be called either, so the session has nothing populated for the answer. We should have some kind of sentinel auto-injected here that can be displayed as output rather than just an empty string for this case.

mskarlin avatar Nov 26 '24 17:11 mskarlin

Yeah to add an example:

import asyncio

from paperqa.litqa import LitQAEvaluation

qa_prompt2, eval_fn2 = LitQAEvaluation.from_question(
    ideal="circular dichroism",
    distractors=["cryo EM", "x-ray crystallography", "NMR"],
    question=(
        "What method was used to demonstrate that the enzyme PafA is stable after"
        " incubation with 4M urea for 14 days?"
    ),
)
print(asyncio.run(eval_fn2("")))  # correct

I am actually working on this right now

The issue is without an answer, the LLM begins to pull on its innate knowledge

jamesbraza avatar Nov 26 '24 18:11 jamesbraza