How to disable model cost computation?

[gdurif]

Hi,

I am running paper-qa with a local LLM served by ollama server, following configuration described in #582. I get several message Could not find cost for model ollama/llama3.2 which I assume is the inference cost in dollars given in the output:

Status: Paper Count=1 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000

Is there a way to disable this cost computation?

Thanks, Best

[dosubot[bot]]

I couldn't find any specific configuration option in the paper-qa codebase to disable cost computation for models. The cost computation is implemented in the add_tokens method of the paperqa/types.py file, where it updates the token counts and adds the cost from the result to the self.cost attribute ^[1]. You might need to modify the code directly to bypass or disable this functionality.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[jamesbraza]

Hi @gdurif the costs come from litellm.model_cost at https://github.com/BerriAI/litellm/blob/v1.60.6/litellm/init.py#L310. It looks like ollama/llama3.2 isn't in their mapping yet, it would be worth adding this there: https://models.litellm.ai/

---

Note the agent actually sees this cost information, and I agree if it's never updated beyond $0, then it's not useful.

If you are programmatically using paper-qa via Python, then you can pass a custom status_fn attribute to EnvironmentState: https://github.com/Future-House/paper-qa/blob/v5.11.1/paperqa/agents/tools.py#L54-L61

However, that may be hard to figure out because it bypasses the agent provisioning logic in paperqa.agents.main. Unfortunately as of v5.11.1 we don't expose any status configuration via Settings, which arguably we should.

I will leave this open for us to add or create something.

[gdurif]

Hi @jamesbraza Thanks for the detailed answer.