feat: add support to only reply when tagged in thread

[Thunderbottom]

Adds an optional flag MATTERMOST_REPLY_ON_TAG which prevents the bot from replying when not tagged in a thread

This also prevents the thread from being hijacked by GPT, since it replies to every single thread post after being mentioned once

[yGuy]

Thanks!

I understand the problem, but as a global setting isn't it pretty annoying to have to "@"mention the bot in every response when you really want a dialog?

Can we come up with a more clever logic? Like only reply automatically if there's only the bot and another user in the thread or when the previous bot reply is the second last reply and happened shortly before that. We could even ask chatgpt whether it thinks that it is "expected" to answer. We could come up with a prompt that would enable the bot to not answer when it "understands" that it isn't even questioned?

I would much prefer a more granular setting or behavior. Any ideas?

[Thunderbottom]

I think for starters, the current solution is pretty simple and practical. In scenarios where mattermost is used (assuming it is mostly in teams at workplaces), I think it's better to be explicitly calling the bot with the username rather than having it reply after every message.

Like only reply automatically if there's only the bot and another user in the thread

This kind of sounds plausible, but then it also means that the bot will only reply when there are two people in the conversation, including the bot itself, rendering it useless in cases where there's multiple people on the thread. Now in such case, you'd suggest triggering the bot again with the username. Having extra logic to end up doing the same thing kind of defeats the purpose, no?

or when the previous bot reply is the second last reply and happened shortly before that.

This I believe means that the bot will reply to every single message after the bot sends something, which is kind of the same issue we've been facing so far with spam replies by the bot.

We could even ask chatgpt whether it thinks that it is "expected" to answer.

Right now, all the thread messages are sent to the API and a response is returned regardless. Practically speaking, by adding this suggested behavior we're only going to see an increase in costs towards the API (the prices are based on number of prompt tokens), and we'll end up having responses only when it feels that it is expected to respond.

We could come up with a prompt that would enable the bot to not answer when it "understands" that it isn't even questioned?

My response to this would still be the same as above, any request we make to the OpenAI API would cost us money, and in this case, be wasted on responses that we never get from it.

Let me know what you think of this :)

[yGuy]

Ok, thanks for your comments and thoughts. Let's come up with a set of use-cases and consider the alternatives.

The alternatives would be:

• current behavior: as soon as the bot was invited to a thread, it will always respond to every new comment
• the bot will only ever respond to a message in which it was directly mentioned - that would effectively remove all thread logic, wouldn't it?
• the bot or the plugin tries to be smart about when the bot should get involved:
  • when there is only one human who can see the response, it could provide that response, even if it's not asked. So once it is invited, it will participate, until someone else gets invited (and thus the two humans want to be undisturbed) - we could even have the bot say "good-bye, have fun!" and then only reactivate when it is spoken to.
• come up with a "shut up bot" magic word, that serves as a stop-word and the bot will be silent until it is mentioned again

And more.

Looking back at my (direct) conversations with the bot, I would find it super annoying to have to add @chatgpt to each of my responses. I sometimes even forget it for the first post and that annoys me, already.

And using the chat bot for "one-shot" questions only, kind of defeats the purpose of the "chat" feature.

My favorite currently would be to always respond to direct communication (dialogs with the bot and one user), and then back off when there are more users in the thread until someone invites it again, in combination with a "stop-word" that renders the bot silent.

What do you think?

[Thunderbottom]

Looking back at my (direct) conversations with the bot, I would find it super annoying to have to add @chatgpt to each of my responses. I sometimes even forget it for the first post and that annoys me, already.

Over the existing enhancement from this PR, we could check if the conversation is a DM or a channel, and continue the existing behavior over DM and keep the "don't reply unless tagged" for channels.

In my perspective from what we've seen so far, we invoke GPT in-between a thread to answer a query and no longer need it after it's done. So invoking it again explicitly when required is good enough for our use case, since it otherwise spams the thread and renders it useless with the current behavior.

My favorite currently would be to always respond to direct communication (dialogs with the bot and one user), and then back off when there are more users in the thread until someone invites it again, in combination with a "stop-word" that renders the bot silent.

This is kind of hard to implement. What you're currently suggesting is that the thread is active with people messaging constantly all the time, which is not really the case a lot of times. And currently each websocket connection message triggers an API request, which we'll need to somehow buffer over, let's say, x seconds. In cases where a person replies on a thread and is not met by a human reply within the next x seconds, ChatGPT will respond. Practically speaking from my perspective, it is kind of hard to determine when another person will reply. And given a workspace scenario, I doubt people would appreciate having unsolicited responses from ChatGPT when they least expect it (i.e. triggered after x seconds of no response).

Quick Edit: For the part where you enter a stop word to trigger ChatGPT to stop functioning, wouldn't it be rather better to invoke it when required, rather than asking it to stop?

I still believe the current solution is good enough, and we could implement the case where it checks whether the current communication is a DM or a channel/group, and acts accordingly.

Let me know what your thoughts are on this.

[yGuy]

Thanks - regarding the timing: for one, we are awaiting the chatgpt response anyways which can take many many seconds. I don't think waiting for 3 more seconds or so would cause any issues. So I don't think there are any technical hurdles.

Regarding the "inverse stop word", I think you are correct. How about this:

• if the bot is mentioned, answer directly
• if the bot is not mentioned, but the conversation so far is limited to the bot and the one who posted the last message, respond, directly. (note that currently we are only looking at the messages within the last 24 hours or so, we could make this attention span configurable, too). (this is slightly more than the direct chat, which I think is a must to support)
• if someone else enters the thread: say "good bye, let me know when you need me!" (optional, maybe configurable, too), and then only "activate" when someone mentions the bot again
• optionally implement a stop word that sends the bot to "inactive" mode for the current thread, even if it was "active" according to the above rules. How about "Thanks @chatgpt."?

I think that would be a very good improvement. So I would be happy to review and merge an implementation/pull-request.

[Thunderbottom]

if someone else enters the thread: say "good bye, let me know when you need me!" (optional, maybe configurable, too), and then only "activate" when someone mentions the bot again

This again I believe is kind of spam that the bot will send over the chat, and we'll have to re-activate it every time a new user shows up in the thread, which is kind of like explicitly triggering the bot with it's username.

optionally implement a stop word that sends the bot to "inactive" mode for the current thread, even if it was "active" according to the above rules. How about "Thanks @chatgpt"?

Wouldn't it be the same thing as this PR, but the opposite? Instead of triggering it with it's username, we'll just end up triggering it with the stopword. Additionally, we'll also need to keep track of when the words "Thanks chatgpt" have been mentioned.

From my perspective, I still think that the only other practical solution would be to allow unlimited replies over DM, and keep it limited to the trigger word on the channel. If need be, we can add a check to see if there's only two people in the thread on the channel as well, and then if more people join we can use the "don't reply unless tagged". That should be easy enough to do.

Any stopwords, keywords, implicitly letting ChatGPT decide would only add to the costs and the code-effort to maintain the state of current thread.

Let me know what you think of this. I think it should be good middle-ground for most use-cases and reduce the spam by a lot as well.

[yGuy]

This again I believe is kind of spam that the bot will send over the chat, and we'll have to re-activate it every time a new user shows up in the thread, which is kind of like explicitly triggering the bot with it's username.

That would only be triggered, once, of course, when we switch from a conversation where the bot "is part of" to when the bot "leaves" the conversation. I don't see how this would be SPAM if it only happens once at the end during a conversation.

But anyways, this can be debated, so I added the "optional" in there.

Wouldn't it be the same thing as this PR, but the opposite?

Yes, same but the opposite :-D

Instead of triggering it with it's username, we'll just end up triggering it with the stopword. Additionally, we'll also need to keep track of when the words "Thanks chatgpt" have been mentioned.

Do you think that the tracking would be hard to implement? I don't understand the reasoning here. My wish is to get the best user experience. If this requires more code, I'm happy. I would not sacrifice usability for simpler code (requiring the user during a dialog to mention "@chatgpt" in every reply is a major annoyance, IMHO. That's why I am trying to find better alternatives).

Any stopwords, keywords, implicitly letting ChatGPT decide would only add to the costs and the code-effort to maintain the state of current thread.

So what? We can add all kind of meta data to all replies/posts and we need to check them anyway. If it improves usability I am all for it.

If you are looking for a simple hack to change the behavior, I'm fine with that, but I would not want to have it in this repo. I don't want to add an option that solves one problem but adds to others. If you don't like that, feel free to fork the repo, but I would really like to improve the situation for all users, and not just one use-case.

So to make this even more explicit: I will accept the pull request, maybe even make it the default behavior, if you make it possible to chat with the bot in a thread where there is only one user plus the bot (direct messages, and threads) without requiring the user to always mention the bot in every sentence. Because this would really SPAM the conversation, IMHO.

[Thunderbottom]

Sorry I couldn't check since I've been traveling.

I will accept the pull request, maybe even make it the default behavior, if you make it possible to chat with the bot in a thread where there is only one user plus the bot (direct messages, and threads) without requiring the user to always mention the bot in every sentence.

That is what I have been saying too :slightly_smiling_face: Let me see what I can do about this and update this PR

[Thunderbottom]

I got it to work in DMs without requiring it to be tagged at all, including threads.

It'll still require for the bot to be tagged in group threads, since I could not find anything in the websocket that listed the number of people participating.

[your-highness]

Any idea when this will be merged? It's a crucial feature in threads.

[yGuy]

I haven't found the time to properly review it. I am not convinced this will be a good user experience and would like to see this discussed in depth, first. I feel that it's either inconvenient or insufficient. The implementation is fine but the idea isn't well thought through IMHO. Has anybody been running this implementation in production and gathered feedback?