hibobmaster
Robots do not respond to messages with "!v"
"gpt_vision_api_endpoint": "https://xxxx/v1/chat/completions", "gpt_vision_model": "gpt-4-turbo",
"content": {
"body": "!v Please explain the above diagram.",
"format": "org.matrix.custom.html",
"formatted_body": "!v Please explain the above diagram.",
"msgtype": "m.text"
}
No response, no logs.
You should quote a image. Besides, gpt vision won't work in E2EE room.
I use @+!v, and it responds twice; the first time, it indicates that it doesn't know.
Like thread level chatting for element android before, !v gpt vision command won't work...
No m.mentions from element android cause a lot of inconvinient things.
https://github.com/hibobmaster/matrix_chatgpt_bot/blob/90a00c9b5aa95cdc2e29f1b6eefe31ebcf3c1e69/src/bot.py#L378-L386
I know what's wrong. When we mention the bot, it trigger thread chat as the same time.
I know what's wrong. When we mention the bot, it trigger thread chat as the same time.
Perhaps the command "!v" can be omitted. Instead, different models could be invoked based on the event's mimetype, as some models, such as gpt-4-turbo and claude-3, support vision.
With https://github.com/hibobmaster/matrix_chatgpt_bot/commit/81543d561b46df4158892324172b5145e44f0e32, mention bot
- with image will trigger gpt vision
- with plain text will trigger thread level chatting
Try image: hibobmaster/matrixchatgptbot:sha-81543d561b46df4158892324172b5145e44f0e32
sha-81543d561b46df4158892324172b5145e44f0e32
Robot can now recognize images in rooms without !v, but they cannot perform this function within threads. Additionally, commands like !pic, !help, and !lc are also unusable in threads, indicating that the two interaction modes are not well integrated. Of course, commands such as !gpt, !chat, and !new are unnecessary in threads. However, due to the poor compatibility of "io.element.thread" (Element PC's implementation for servers that do not support the Matrix standard threads) on client devices (it does not display on mobile phones), retaining these commands is still essential.
With 81543d5, mention bot
- with image will trigger gpt vision
- with plain text will trigger thread level chatting
Try image:
hibobmaster/matrixchatgptbot:sha-81543d561b46df4158892324172b5145e44f0e32
Some of the robot's responses are displayed entirely in red font, while others are not. It appears that the <mx-reply> <blockquote> tags have been added in the HTML. Is this another way of implementing the reply function in Matrix? Very strange!
https://github.com/hibobmaster/matrix_chatgpt_bot/commit/c5834db9b270181a9987aff05c311d6c698a3d49
Try image: hibobmaster/matrixchatgptbot:sha-c5834db9b270181a9987aff05c311d6c698a3d49
sha-c5834db9b270181a9987aff05c311d6c698a3d49
I tested it, and these commands execute successfully within threads. They seem to be independent of the thread context. For example, after uploading a picture, the bot cannot describe it directly. Instead, it requires a separate reply to reference it, and the bot's response is not considered part of the thread context. Of course, the current method is okay, almost like a thread within a thread 😀. This can reduce unnecessary context, which can be referenced separately when needed.
the bot cannot describe it directly
GPT Vision has a prompt which should be provided by user since matrix doesn't support sending a image with description.
and the bot's response is not considered part of the thread context
Try image: hibobmaster/matrixchatgptbot:v1.7.2
the bot cannot describe it directly
GPT Vision has a prompt which should be provided by user since matrix doesn't support sending a image with description.
and the bot's response is not considered part of the thread context
Try image:
hibobmaster/matrixchatgptbot:v1.7.2
done👍