facebookresearch
Multiple Users Using Parlai Chat Service
Hi, my task is to develop a service where multiple users can chat with blenderbot2, here's a quick overview of my general setup.
Entry Point: I'm using the websocket chat service as the entry point. I made the same modifications to socket.py as the person who posted this issue so that the sid is set based on a user id that's passed in.
World: My world file is pretty much the same as the demo one here. A new world is created for each human agent that enters a chat and a new blenderbot agent is cloned. I am not passing in any message history so all conversation history is being handled by the blenderbot agent.
Config Here is my config file:
tasks:
default:
onboard_world: MessengerBotChatOnboardWorld
task_world: MessengerBotChatTaskWorld
timeout: 1800
agents_required: 1
task_name: chatbot
world_module: parlai.chat_service.tasks.chatbot.worlds
overworld: MessengerOverworld
max_workers: 100
opt:
debug: True
models:
blenderbot2_3B:
model: projects.blenderbot2.agents.blenderbot2:BlenderBot2FidAgent
model_file: zoo:blenderbot2/blenderbot2_3B/model
interactive_mode: True
no_cuda: False
override:
search_server: https://www.google.com
additional_args:
page_id: 1 # Configure Your Own Page
My issue:
The issue I'm running into occurs when two users send a message to the bot simultaneously. Whenever this happens, one of the user's chats breaks and I get this error message:
World default had error RuntimeError('Last observe() was episode_done, but we never saw a corresponding self_observe to clear the history, probably because you missed an act(). This was changed in #2043. File a GitHub issue if you require assistance.')
This only happens when users send messages simultaneously, if one user sends a message then another user sends one after a delay I get no errors and the bot agent's memories/conversation history is unaffected. The delay doesn't necessarily have to be long enough for the bot to finish responding to the first user, it seems to only happen during one of the initial response phases.
Additional context: I managed to stop this error by adding these lines at the start of the observe() function in torch_agent.py:
if observation.get('episode_done'):
observation.force_set('episode_done', False)
but this causes the bot's persona memories to get messed up (some of the persona memories start looking like this: __NO__PERSONA__BEAM__MIN__LEN__20__) although the conversation histories don't seem to be affected.
What doesn't make sense to me is why you're getting 'episode_done': True in the first place. What would two agents messaging have to do with that?
That persona generated is not unexpected --> it means that you should not write any new memories to the memory store.
I don't think that I'm getting 'episode_done': True as a result of two agents messaging simultaneously. I added some print statements to observe() in torch_agent.py and it seems like observe() is called multiple times during the agents response. The first time episode done is False bc that's what I pass in initially but on the subsequent calls it's True for some reason. Here is the command line output when I include the print statement:
===act====
{'episode_done': False, 'text': 'Hi', 'payload': None}
~~~~~~~~~~~
episode_done: False
episode_done: True
episode_done: True
episode_done: True
/Users/ryanshea/Documents/parlai/parlai/core/torch_generator_agent.py:1610: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
hyp_ids = best_idxs // voc_size
episode_done: True
15:12:45 ERRO | Failed to retrieve data from server! Search server returned status 405
15:12:45 WARN | Server search did not produce any results for "EduBot" query. returning an empty set of results for this query.
BlenderBot's history view:
Hello!
Hello! This is EduBot! How are you doing today?
Hi
Hi! I am EduBot. I am a bot that answers questions.
I think that cross user interference is happening during some phase of the agent's response. I'm not really sure why though bc I'm cloning a new agent each time a new world is generated.
Any thoughts on this? Let me know if there's any other information I can provide to clarify the problem.
I think your print statements are capturing calls to the search query generation and memory generation modules. Could you try including the logging and pasting the full stack trace with your above error? it's possible that those modules are not being "shared"/cloned appropriately
Sure, here is the log file: 1658768422.865366.log
And here is the stack trace:
World default had error RuntimeError('Last observe() was episode_done, but we never saw a corresponding self_observe to clear the history, probably because you missed an act(). This was changed in #2043. File a GitHub issue if you require assistance.')
World default had error RuntimeError('Last observe() was episode_done, but we never saw a corresponding self_observe to clear the history, probably because you missed an act(). This was changed in #2043. File a GitHub issue if you require assistance.')
Traceback (most recent call last):
File "/Users/ryanshea/opt/anaconda3/envs/parlai_blender2/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/ryanshea/Documents/parlai/parlai/chat_service/core/world_runner.py", line 128, in _world_fn
return self._run_world(task, world_name, agents)
File "/Users/ryanshea/Documents/parlai/parlai/chat_service/core/world_runner.py", line 99, in _run_world
ret_val = world.parley()
File "/Users/ryanshea/Documents/parlai/parlai/chat_service/tasks/chatbot/worlds.py", line 106, in parley
response = self.model.act()
File "/Users/ryanshea/Documents/parlai/parlai/core/torch_agent.py", line 2146, in act
response = self.batch_act([self.observation])[0]
File "/Users/ryanshea/Documents/parlai/parlai/core/torch_agent.py", line 2242, in batch_act
output = self.eval_step(batch)
File "/Users/ryanshea/Documents/parlai/projects/blenderbot2/agents/blenderbot2.py", line 792, in eval_step
output = super().eval_step(batch)
File "/Users/ryanshea/Documents/parlai/parlai/agents/rag/rag.py", line 290, in eval_step
output = super().eval_step(batch)
File "/Users/ryanshea/Documents/parlai/parlai/core/torch_generator_agent.py", line 875, in eval_step
beam_preds_scores, beams = self._generate(
File "/Users/ryanshea/Documents/parlai/parlai/agents/rag/rag.py", line 673, in _generate
gen_outs = self._rag_generate(batch, beam_size, max_ts, prefix_tokens)
File "/Users/ryanshea/Documents/parlai/parlai/agents/rag/rag.py", line 712, in _rag_generate
return self._generation_agent._generate(
File "/Users/ryanshea/Documents/parlai/parlai/core/torch_generator_agent.py", line 1094, in _generate
encoder_states = model.encoder(*self._encoder_input(batch))
File "/Users/ryanshea/Documents/parlai/projects/blenderbot2/agents/modules.py", line 842, in encoder
enc_out, mask, input_turns_cnt, top_docs, top_doc_scores = super().encoder( # type: ignore
File "/Users/ryanshea/Documents/parlai/projects/blenderbot2/agents/modules.py", line 217, in encoder
expanded_input, top_docs, top_doc_scores = self.retrieve_and_concat(
File "/Users/ryanshea/Documents/parlai/projects/blenderbot2/agents/modules.py", line 345, in retrieve_and_concat
generated_memories = self.memory_decoder.generate_memories(
File "/Users/ryanshea/Documents/parlai/projects/blenderbot2/agents/sub_modules.py", line 322, in generate_memories
raw_memories_i = list(reversed(self._batch_generate(context_lines)))
File "/Users/ryanshea/Documents/parlai/projects/blenderbot2/agents/sub_modules.py", line 142, in _batch_generate
agent_i.observe(Message({'text': t_i, 'episode_done': True}))
File "/Users/ryanshea/Documents/parlai/parlai/core/torch_agent.py", line 1836, in observe
self._validate_observe_invariants()
File "/Users/ryanshea/Documents/parlai/parlai/core/torch_agent.py", line 1948, in _validate_observe_invariants
raise RuntimeError(
RuntimeError: Last observe() was episode_done, but we never saw a corresponding self_observe to clear the history, probably because you missed an act(). This was changed in #2043. File a GitHub issue if you require assistance.
Does this clarify anything? Is there any other method I could use to clone the agent so that those sub modules are shared properly?
sorry yes it absolutely does, thank you for sharing the full stack trace. The gist of the issue is that we use batching in the memory decoder / search query generator, if there are batched inputs, but for now we assume that the batchsize remains the same over the lifetime of the agent (which is not the case if you have several people sending messages at varying intervals)
The lines here should instead be some form of the following:
for agent_i, t_i in zip(active_agents, texts):
try:
agent_i.observe(Message({'text': t_i, 'episode_done': True}))
except RuntimeError:
logging.debug(f"agent {agent_i} need not observe episode done")
continue
Thanks for the response. That got rid of the error (I also had to change these lines though) however it seems to be causing some issues with the blenderbot personas/search queries. I've pasted some logs below, in the blenderbot response you can see that the first agent's persona was written to the second agent and the search query for the second agent seems to be incorrect. Any idea what could be causing this? I ran into a similar issue when I tried to force set episode done to false in torch_agent.py.
[I 220802 11:11:21 web:2271] 101 GET /websocket (::1) 16.49ms
[I 220802 11:11:21 sockets:60] websocket message from client: b'{"text": "I play tennis.", "user_id": "1234"}'
[I 220802 11:11:21 sockets:65] Current subscribers:
[I 220802 11:11:21 sockets:66] Changed sid to 1234
[I 220802 11:11:21 agents:55] Received new message: {'text': 'I play tennis.', 'payload': None, 'sender': {'id': '1234'}, 'recipient': {'id': 0}}
===act====
{'episode_done': False, 'text': 'I play tennis.', 'payload': None}
~~~~~~~~~~~
[I 220802 11:11:22 web:2271] 101 GET /websocket (::1) 2.46ms
[I 220802 11:11:22 sockets:60] websocket message from client: b'{"text": "I\'m from New York.", "user_id": "123"}'
[I 220802 11:11:22 sockets:65] Current subscribers:
[I 220802 11:11:22 sockets:66] Changed sid to 123
[I 220802 11:11:22 agents:55] Received new message: {'text': "I'm from New York.", 'payload': None, 'sender': {'id': '123'}, 'recipient': {'id': 0}}
===act====
{'episode_done': False, 'text': "I'm from New York.", 'payload': None}
~~~~~~~~~~~
11:11:54 ERRO | Failed to retrieve data from server! Search server returned status 405
11:11:54 ERRO | Failed to retrieve data from server! Search server returned status 405
11:11:54 WARN | Server search did not produce any results for "tennis" query. returning an empty set of results for this query.
11:11:54 WARN | Server search did not produce any results for "tennis" query. returning an empty set of results for this query.
===response====
{'id': 'BlenderBot2Fid', 'episode_done': False, 'text': 'Tennis is a great sport. Do you like to play it too? I love to play.', 'beam_texts': [('Tennis is a great sport. Do you like to play it too? I love to play.', -11.756305694580078), ('Tennis is a great sport. Do you like to play it too? I love playing it.', -11.815750122070312), ('Tennis is a great sport. Do you like to play it too? I play tennis too.', -11.914169311523438), ('Tennis is a great sport. Do you like to play it too? I like to watch it.', -12.082024574279785), ('Tennis is a great sport. Do you like to play it too? I like to watch.', -12.147582054138184), ('Tennis is a great sport. Do you like to play it too? I like to watch it. ', -13.069994926452637), ('Tennis is a great sport. Do you like to play it too? I love playing it. _', -13.973596572875977), ('Tennis is a great sport. Do you like to play it too? I love to play. _', -13.995917320251465), ('Tennis is a great sport. Do you like to play it too? I play tennis too. _', -14.118517875671387), ('Tennis is a great sport. Do you like to play it too? I like to watch it. _', -14.130386352539062)], 'top_docs': ['_0 | ', '_0 | ', '_0 | ', '_0 | ', '_0 | '], 'search_queries': 'tennis', 'memories': ["partner's persona: I play tennis. I am female. I like tennis."], 'metrics': {'clen': AverageMetric(50), 'ctrunc': AverageMetric(0), 'ctrunclen': AverageMetric(0)}}
~~~~~~~~~~~
[I 220802 11:12:36 agents:40] Sending new message: {'id': 'BlenderBot2Fid', 'episode_done': False, 'text': 'Tennis is a great sport. Do you like to play it too? I love to play.', 'beam_texts': [('Tennis is a great sport. Do you like to play it too? I love to play.', -11.756305694580078), ('Tennis is a great sport. Do you like to play it too? I love playing it.', -11.815750122070312), ('Tennis is a great sport. Do you like to play it too? I play tennis too.', -11.914169311523438), ('Tennis is a great sport. Do you like to play it too? I like to watch it.', -12.082024574279785), ('Tennis is a great sport. Do you like to play it too? I like to watch.', -12.147582054138184), ('Tennis is a great sport. Do you like to play it too? I like to watch it. ', -13.069994926452637), ('Tennis is a great sport. Do you like to play it too? I love playing it. _', -13.973596572875977), ('Tennis is a great sport. Do you like to play it too? I love to play. _', -13.995917320251465), ('Tennis is a great sport. Do you like to play it too? I play tennis too. _', -14.118517875671387), ('Tennis is a great sport. Do you like to play it too? I like to watch it. _', -14.130386352539062)], 'top_docs': ['_0 | ', '_0 | ', '_0 | ', '_0 | ', '_0 | '], 'search_queries': 'tennis', 'memories': ["partner's persona: I play tennis. I am female. I like tennis."], 'metrics': {'clen': AverageMetric(50), 'ctrunc': AverageMetric(0), 'ctrunclen': AverageMetric(0)}}
BlenderBot's history view:
Hello!
Hello! This is EduBot! How are you doing today?
Hi
Hi! I am EduBot. I am a bot that answers questions. _POTENTIALLY_UNSAFE__
I play tennis.
Tennis is a great sport. Do you like to play it too? I love to play.
===response====
{'id': 'BlenderBot2Fid', 'episode_done': False, 'text': "Hi, I'm EduBot, and I'm a bot. I answer questions. What about you?", 'beam_texts': [("Hi, I'm EduBot, and I'm a bot. I answer questions. What about you?", -9.815694808959961), ("Hi, I'm EduBot, and I'm from the United States. What about you? Where are you from?", -9.85122299194336), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have any questions?", -9.908370018005371), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have any questions for me?", -10.157482147216797), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have a question?", -10.246025085449219), ("New York is a great place to live. It's a great city to live in. What do you do for work?", -10.36941909790039), ("New York is a great place to live. It's a great city to live in. What do you like to do?", -10.383301734924316), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have any?", -10.611528396606445), ("Hi, I'm EduBot, and I'm from the United States. What about you? ", -10.653157234191895), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have questions?", -10.856289863586426)], 'top_docs': ['_0 | ', '_0 | ', '_0 | ', '_0 | ', '_0 | '], 'search_queries': 'tennis', 'memories': ["partner's persona: I play tennis. I am female. I like tennis."], 'metrics': {'clen': AverageMetric(52), 'ctrunc': AverageMetric(0), 'ctrunclen': AverageMetric(0)}}
~~~~~~~~~~~
[I 220802 11:12:38 agents:40] Sending new message: {'id': 'BlenderBot2Fid', 'episode_done': False, 'text': "Hi, I'm EduBot, and I'm a bot. I answer questions. What about you?", 'beam_texts': [("Hi, I'm EduBot, and I'm a bot. I answer questions. What about you?", -9.815694808959961), ("Hi, I'm EduBot, and I'm from the United States. What about you? Where are you from?", -9.85122299194336), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have any questions?", -9.908370018005371), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have any questions for me?", -10.157482147216797), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have a question?", -10.246025085449219), ("New York is a great place to live. It's a great city to live in. What do you do for work?", -10.36941909790039), ("New York is a great place to live. It's a great city to live in. What do you like to do?", -10.383301734924316), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have any?", -10.611528396606445), ("Hi, I'm EduBot, and I'm from the United States. What about you? ", -10.653157234191895), ("Hi, I'm EduBot, and I'm a bot. I answer questions. Do you have questions?", -10.856289863586426)], 'top_docs': ['_0 | ', '_0 | ', '_0 | ', '_0 | ', '_0 | '], 'search_queries': 'tennis', 'memories': ["partner's persona: I play tennis. I am female. I like tennis."], 'metrics': {'clen': AverageMetric(52), 'ctrunc': AverageMetric(0), 'ctrunclen': AverageMetric(0)}}
BlenderBot's history view:
Hello!
Hello! This is EduBot! How are you doing today?
Hi
Hi! I am EduBot. I am a bot that answers questions. _POTENTIALLY_UNSAFE__
I'm from New York.
Hi, I'm EduBot, and I'm a bot. I answer questions. What about you?
something else appears to be going wrong with the agent cloning, then, if they are sharing history.
which lines are you printing this in? I think maybe we should just call agent_i.reset() above and don't even bother with the self observe that you linked to
I'm printing this in the worlds file whenever the world receives a message from the user. I tried reseting the model agent in the world before it observes/acts (inserting self.model.reset() above this line) as well as changing episode_done to True but neither has worked. I think the interference comes from when two agents try to write to their persona memories at the same time because it seems like removing the error by either force setting episode_done or doing the try catch messes with the personas. Is there a way to adjust the way the agents are cloned or something to properly separate their persona memories?
Perhaps something is going wrong with the batching - might it be possible to wrap your calls to the model in a semaphore such that only one user has access to the model at a time?
I see, that is actually my current implementation. I wanted to see if calls could be done concurrently but it seems like getting that to work may be quite difficult. Either way this issue seems to be resolved in BlenderBot 3 so I can just switch to using that one. Thanks for the help with everything
There are a lot of moving parts to BB2, part of the reason why BB3 is the way it is. I'll go ahead and close this issue then, please reopen (or file a new one) if you feel the need to