Issue with Asynchronous Multi-Agent Commands using LMStudio API Causing Incorrect Output Assignment

[mohannahoveyda]

Hi,

Thanks for this useful framework!

There is an issue while running multi-agent commands that require asynchronous use of LMStudio API. For instance, running PYTHONPATH=. python experiments/run_mmlu.py --num-truthful-agents=3 --mode=OptimizedSwarm, the outputs of LMStudio seem to be not assigned to the relevant question but to random input questions.

I have not tested this with Open AI API to see if this is specifically an incompatibility issue of LMStudio.

My current temporary solution is not to use the asynchronous implementation but this is annoyingly very slow. Have you encountered this? Is there a way to still use your asynchronous implementation and not experience this issue?

Thanks so much!