text-generation-webui
text-generation-webui copied to clipboard
oobabooga
Chat with multiple characters at the same time
Description
If possible I'd like to be able to chat with multiple characters simultaneously.
Additional Context
Say, for example I'm in a role play session on the bridge of the USS Enterprise in a Star Trek scenario. There are multiple characters there that I'd like to be able to interact with and that I would like to also interact with each other; preferably the whole bridge crew or more. Do you think this is possible? They could use the same model, just with partially different contexts that describe the different characters. Or is that impossible in the current state of LLMs?
I was working on something like this but couldn't find a good way to automagically decide who would talk next and how to manage who was/wasn't in the "room". I had a slightly more manual version mostly working (you pick who is in the room and the next person to reply is random unless a specific string like "asks <character_name>" was used).
Depending on who is talking, I'd send that character's data, a list of people in the room, and however much of the chat history that would fit into context.
It was pretty janky, and messed up character persistent logs... maybe I'll pick it back up, as I did think it was a fun idea... just need to figure out a better way to manage who is in the room, and who talks next.
I'd like to see such functionality too! But not sure it's easy to implement it.
I had some success getting it to work but not really good. I just used normal names in the character file to define their features and give them some personality and relationships to each other. It would be good if we could get the ability to define more special Tokens in the character files. A Token counter in the character window would tell one if it is just to much info. Also the maximal Token limit needs to be increased somehow. Maybe It would be possible to define different fonts to the different characters. That way the bot could be "Storyteller" and easy to difference. In my success the Bot told some stuff then the additional Charakter use NameX: ..(text)... , NameY: ..(text)... and NameZ: ..(text)...
Project-Akiko support "Chat with multiple characters in character groups"... I think we just need a room with a list of characters. The context for each of them is also saved. Those. if we "remove" the character from the room, he will remember what happened there while he was there. Then, when we "return" it, it will not start with an empty context.
If you are working in a story telling mode on Kobold, the AI has no trouble with multiple characters and deciding when one of them should pipe up. I guess it's just a matter of framing things for the AI and keeping up a good memory of the conversation.
I was working on something like this but couldn't find a good way to automagically decide who would talk next
Just need a "list of characters in the room (including the person)" and use fifo, plus the person can change the position of the character in the list before the next "move"
If you are working in a story telling mode on Kobold, the AI has no trouble with multiple characters and deciding when one of them should pipe up. I guess it's just a matter of framing things for the AI and keeping up a good memory of the conversation.
And UI. The primary chat UI is always between me and the main character. It would be awesome if the UI could reflect multiple characters as well, like in a group chat or something.
Perhaps something similar? On the right we select the characters to be added or removed from the room, on the left we select the one who will speak next.
Yeah, something like that. However, also semi-automatic, like, I could imagine the AI to be capable of determining how likely it would be for one of the characters to interact or not. Like when they are addressed directly or just "want to add their two cents".
I guess though, your solution would be the simplest for now, and since I'm a fan of make it small but good, I'd love to see that implemented.
It could work if you start a question with a name, that character would answer. That one should be really easy. The system would need some gramma rules to be good at this.
I've just tried it a couple times. The AI perfectly understands which person is addressed in a sentence. It would also be interesting to see if AI characters could even interact with each others and not just the human.
The SillyTavern project is attempting group chats, you should have a look. I find they get confused pretty easily, so it is not there yet.
https://github.com/Cohee1207/SillyTavern
That project gives characters personality background, of course, and adds a chattiness slider to determine how likely they are to pipe up in a conversation. So the AI itself isn't deciding if they should talk.
I think you could have a master controlling paragraph sent to the AI with every prompt. It tells the AI who is in the room, who has left, and reminds the AI of the rules of the conversation. Rules like: No character speaks unless it's name is mentioned by the player or another AI. Or characters only speak when prompted like "###Patricia" or something like that. Or a list of character buttons next to the prompt window. If you're addressing a character or specific characters, you turn or leave those buttons on. The software then feeds the appropriate prompt to the AI.
If you can't force the AI to cooperate with this, you'll get a chaotic mess.
I've just tried it a couple times. The AI perfectly understands which person is addressed in a sentence. It would also be interesting to see if AI characters could even interact with each others and not just the human.
It works if you formulate it for certain conditions.
Example:
Spock comments in detail about local flora to Red-shirt and you when you beam down with Spock.
Red-shirt might cry out loud to you if Spock commented about local flora and you beamed down with Spock and they search.
something like this should work. It's important that the chars tell you what they do or you won't be notified.
Right now, a lot of the code that handles chat explicitly refers to user and bot as name1 and name2 respectively:
https://github.com/oobabooga/text-generation-webui/blob/2cf711f35ec8453d8af818be631cb60447e759e2/modules/chat.py#L32-L35
We'll need to come up with a way to replace the current approach with one that allows for multiple bot names, then find and edit everything that assumes the 2-speaker setup. Maybe a list state['names_bot']: list? Separate contexts per character isn't absolutely necessary, but would be nice for combining existing individual cards into a "group chat".
As for how chat history is stored, it's a list of input/reply pairs ['user message', 'bot message']. In the chat handling code, these pairs are indexed explicitly with [0] and [1], assuming a single bot character only:
https://github.com/oobabooga/text-generation-webui/blob/2cf711f35ec8453d8af818be631cb60447e759e2/modules/chat.py#L94-L96
Character names are not stored with the history; instead, they're appended later when needed (e.g. during prompting). In a multi-character setup, we'll need a way to store in the history and determine later which bot character is responsible for which message. This might be difficult to keep backwards-compatible with exported chat histories, since those don't store character names either.
I'll pin this because I don't know how to do it and it seems like an interesting challenge. As @toast22a pointed out, it's not a trivial generalization to make.
This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@St33lMouse thanks for the note on SillyTavern, slightly surprised that ST won't bundle with Oobabooga as the default install (other than KoboldAI of course).
