holoviz
Ability to send text and attachment together in ChatInterface
Description
I have two widgets in my ChatInterface, [pn.widgets.TextInput(...), pn.widgets.FileInput(...)] However I have noticed that I cannot send both a text message and file attachment together.
When I type text and switch to the file input tab, it automatically sends the text message and vice versa.
I would like to achieve this given that LLMs are multimodal and can handle text and images together for example.
Suggested Solution
A solution that would essentially pair text + attachment. The attachment renderers are great. It would be awesome to see a text string + csv uploaded together the UI would reflect a chat message then the rendered mime type. Maybe the underlying contents could be that of a list?
[text, mime_type_attachment] or something to that effect. That way under the hood, say in a callback, when you grab that content to send to an LLM you can distinguish the text string vs the attachment.
Considered alternatives
I have tried to implement something like this manually through callbacks and such, but can't quite get it to look as good as the renderers do nor be able to handle them together as one message.
Additional context
https://discourse.holoviz.org/t/ability-to-send-both-text-and-file-attachement-intput-together-in-chatinterface/8742
Something like this, where I send a text but I also attach a CSV. They could be published as the same message or two consecutive, whatever works.
But then on the backend of things the input is some sort of object that pairs them together, like
[text, attachment] | {"text": <input>, "attachment": <mime_type>} | etc.
This issue has been mentioned on HoloViz Discourse. There might be relevant details there:
https://discourse.holoviz.org/t/ability-to-send-both-text-and-file-attachement-intput-together-in-chatinterface/8742/3
To address this, I think it'd be good to have a FileChatAreaInput that combines ChatAreaInput with a FileDropper overlaid, like how most sites work.
In the mean time, maybe you could combine FileDropper, TextAreaInput, and Button with https://panel.holoviz.org/reference/custom_components/PyComponent.html
In the mean time, maybe you could combine FileDropper, TextAreaInput, and Button with https://panel.holoviz.org/reference/custom_components/PyComponent.html
Would you possibly be able to share a minmal repro? No worries if not - may just help me get started a bit! @ahuang11
Something like:
import param
import panel as pn
from panel.custom import PyComponent
from panel.chat.interface import ChatInterface
pn.extension()
class MessageFileSender(PyComponent):
value = param.Parameter()
def __init__(self, **params):
super().__init__(**params)
self.text_input = pn.widgets.TextAreaInput(placeholder="Type your message here...", sizing_mode="stretch_width", name=" ")
self.file_input = pn.widgets.FileDropper(sizing_mode="stretch_width")
pn.bind(self._update_value, self.text_input, self.file_input, watch=True)
self._layout = pn.Tabs(
("Text", self.text_input),
("File", self.file_input),
)
def _update_value(self, text, file):
print("text", text)
print("file", file)
if file:
file_panel = pn.panel(list(file.values())[0])
self.value = pn.Column(file_panel, text)
else:
self.value = text
def __panel__(self):
return self._layout
component = MessageFileSender()
ChatInterface(widgets=[component]).show()
Something like:
import param import panel as pn from panel.custom import PyComponent from panel.chat.interface import ChatInterface
pn.extension()
class MessageFileSender(PyComponent): value = param.Parameter()
def __init__(self, **params): super().__init__(**params) self.text_input = pn.widgets.TextAreaInput(placeholder="Type your message here...", sizing_mode="stretch_width", name=" ") self.file_input = pn.widgets.FileDropper(sizing_mode="stretch_width") pn.bind(self._update_value, self.text_input, self.file_input, watch=True) self._layout = pn.Tabs( ("Text", self.text_input), ("File", self.file_input), ) def _update_value(self, text, file): print("text", text) print("file", file) if file: file_panel = pn.panel(list(file.values())[0]) self.value = pn.Column(file_panel, text) else: self.value = text def __panel__(self): return self._layoutcomponent = MessageFileSender()
ChatInterface(widgets=[component]).show()
I see the following error using this component with a servable app:
AttributeError: 'MessageFileSender' object has no attribute '_process_param_change'
Traceback (most recent call last):
File "/<>/panel/io/handlers.py", line 412, in run
exec(self._code, module.__dict__)
File "<>/app.py", line 411, in <module>
app.run()
File "<>/app.py", line 407, in run
return app.servable()
File "/<>/panel/template/base.py", line 543, in servable
return super().servable(title, location, area, target)
File "/<>/panel/viewable.py", line 416, in servable
self.server_doc(title=title, location=location) # type: ignore
File "/<>panel/template/base.py", line 505, in server_doc
return self._init_doc(doc, title=title, location=location)
File "/<>/panel/template/base.py", line 761, in _init_doc
document = super()._init_doc(doc, comm, title, notebook, location)
File "/<>panel/template/base.py", line 266, in _init_doc
self._design.apply(col, preprocess_root, isolated=False)
File "/<>/panel/theme/base.py", line 342, in apply
self._reapply(viewable, root, isolated=isolated, cache=cache)
File "/<>/panel/theme/base.py", line 138, in _reapply
self._apply_modifiers(o, ref, self.theme, isolated, cache, document)
File "/<>/panel/theme/base.py", line 254, in _apply_modifiers
cls._apply_params(viewable, mref, modifiers, document)
File "/<>panel/theme/base.py", line 274, in _apply_params
props = viewable._process_param_change(params)
AttributeError: 'MessageFileSender' object has no attribute '_process_param_change'
Any ideas on what is going on here? @ahuang11
Oh... looks like it needed to inhereit both:
class MessageFileSender(PyComponent, pn.widgets.CompositeWidget):