Effect of batch size on answer quality and speed.

[Snikch63200]

Hello,

I changed batch size from 1 (default) to 8, 32 and saw no changes on paperQA behavioural (answer quality end speed), as follows :

settings=Settings(
    llm=f"openai/mixtral:8x7b",
    llm_config=local_llm_config,
    summary_llm=f"openai/mixtral:8x7b",
    summary_llm_config=local_llm_config,
    verbosity=3,
    temperature=0.7,
    batch_size=32,
    index_directory="indexes",
    paper_directory="./Papers",
    agent=AgentSettings(
                        agent_llm_config=local_llm_config,
                        agent_llm=f"openai/mixtral:8x7b",
                        agent_type="ToolSelector",
                        ),
    )

Is batch size have any impact on paperQA behavioural ?

[dosubot[bot]]

The batch_size parameter in the paperQA codebase is primarily used for processing text documents in batches during the embedding process. It determines how many documents are processed together in a single batch when calling the embedding function ^[1]. However, there is no specific information available in the codebase that directly links changes in batch_size to variations in answer quality or speed for the openai/mixtral:8x7b model ^[1].

The impact of batch_size might not be noticeable in terms of answer quality and speed, especially if the bottleneck is elsewhere in the system or if the model and hardware can handle larger batches efficiently.

To continue talking to Dosu, mention @dosu.