paper-qa icon indicating copy to clipboard operation
paper-qa copied to clipboard

Broken title search ut

Open nadolskit opened this issue 1 year ago • 1 comments

Summary of Problem The expected vs asserted citation count was not matching, causing this UT to specifically fail. 

I noticed if I switched the order of sources from semantic_scholar, crossref to crossref, semantic_scholar the citation count would be even more dramatically different (similar to the results James was noticing earlier)

More importantly, I found the papers on semantic scholar (25); which doesn't match our expected value (23); so that failure makes sense. The following assertion is failing for the same reason: 196 assertion vs 191 expected.

Vibe checked the results of this change via a CI run: https://github.com/Future-House/paper-qa/actions/runs/10859898582/job/30139826477

This does not fix the issue that James found earlier this evening in test_agents.py.

nadolskit avatar Sep 14 '24 06:09 nadolskit

The minor variations, i.e. 196 vs 191 is expected, because the papers will get more citations over time. The bigger question is why the cassettes aren’t being used because this should come back with a recorded request each time, I’m in favor of root causing that before we merge this. Maybe we regenerated these cassettes but didn’t update the test?

mskarlin avatar Sep 14 '24 16:09 mskarlin