LLamaSharp
LLamaSharp copied to clipboard
SciSharp
Cannot add a user message after another user message (Parameter message
Occasionally appearing "Cannot add a user message after another user message (Parameter message " Suspected to be caused by concurrent calls
Hi, thank you for reporting us it. Could you please provide some steps to reproduce it?
https://github.com/xuzeyu91/AntSK/blob/main/src/AntSK/Services/LLamaSharp/LLamaChatService.cs This is my code, and I use streaming output. If I initiate another request before completing it, this problem will occur
You are trying to use something that is not thread safe, in a multi-threaded scenario. The assistants response does not get added to the ChatSession until the first call to your API is completed, so when a second call comes in, it tries to add the next user message to the ChatSession, before the first one is done, so you end up with two user messages added one after the other. Depending on what your trying to make, it would either make sense to queue up requests and have them wait before added to the chat session, give different chat sessions to each request, or to bypass chat session entirely and use the executor.
Most things only work with a single thread. If you want to share things between requests, they need to be designed to work that way.
Seems to be related with #595. Could you please try again on the current master branch?
Both these methods are not thread safe, it's not going to work reliably under load if they are going through ChatSession, and it's being shared on calls to http requests. I wouldn't consider it a bug though, being thread safe is a feature. ChatHistory, and ChatSession are not even remotely close to thread safe.
https://github.com/SciSharp/LLamaSharp/blob/9ee6ae331918d52f7887e3285361428943fb093b/LLama/ChatSession.cs#L234 https://github.com/SciSharp/LLamaSharp/blob/9ee6ae331918d52f7887e3285361428943fb093b/LLama/ChatSession.cs#L390
Yes, the callings of native apis are thread-safe now but ChatSession doesn't. I misunderstood this issue. Do you have any idea about how to make ChatSession thread-safe?
Making it thread safe doesn't really make sense.
The normal scenario is probably going to be 1 user per 1 chat session, and in that case, it's better to just reject the call until the AI is done chatting. You can see something like this happen if you use ChatGPT, and try to send a message while it's still talking.
Calling Chat could detect that it's in the middle of making a response, and throw an InvalidOperationException while it's still busy, so the error message is clearer. The alternatives of feeding messages to Chat Session in a queue, or interrupting the currently generated message will probably need a custom enough solution that they need to use the executor directly anyways.