langchain
langchain copied to clipboard
Published
20 hours ago •
langchain-ai
langchain-ai
ChatPromtpTemplate doesn't accept PDF data as bytes
Checked other resources
- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
Example Code
pdf_data = base64.b64encode(httpx.get("https://dagrs.berkeley.edu/sites/default/files/2020-01/sample.pdf").content).decode("utf-8")
prompt = ChatPromptTemplate([
("system", "You are a helpful assistant "),
("human", [
{"type": "media", "mime_type": "application/pdf", "data": pdf_data},
{"type": "text", "text": "{user_input}"}
])
])
Error Message and Stack Trace (if applicable)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[4], [line 12](vscode-notebook-cell:?execution_count=4&line=12)
[1](vscode-notebook-cell:?execution_count=4&line=1) model = ChatGoogleGenerativeAI(
[2](vscode-notebook-cell:?execution_count=4&line=2) model="gemini-1.5-flash",
[3](vscode-notebook-cell:?execution_count=4&line=3) temperature=0.1,
(...)
[6](vscode-notebook-cell:?execution_count=4&line=6) max_retries=2
[7](vscode-notebook-cell:?execution_count=4&line=7) )
[9](vscode-notebook-cell:?execution_count=4&line=9) pdf_data = base64.b64encode(
[10](vscode-notebook-cell:?execution_count=4&line=10) httpx.get("http://www.dagrs.berkley.edu/sites/default/files/2020-01/sample.pdf").content).decode("utf-8")
---> [12](vscode-notebook-cell:?execution_count=4&line=12) prompt = ChatPromptTemplate([
[13](vscode-notebook-cell:?execution_count=4&line=13) ("system", "You are a helpful assistant "),
[14](vscode-notebook-cell:?execution_count=4&line=14) ("human", [
[15](vscode-notebook-cell:?execution_count=4&line=15) {"type": "media", "mime_type": "application/pdf", "data": pdf_data},
[16](vscode-notebook-cell:?execution_count=4&line=16) {"type": "text", "text": "{user_input}"}
[17](vscode-notebook-cell:?execution_count=4&line=17) ])
[18](vscode-notebook-cell:?execution_count=4&line=18) ])
File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:992, in ChatPromptTemplate.__init__(self, messages, template_format, **kwargs)
[938](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:938) def __init__(
[939](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:939) self,
[940](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:940) messages: Sequence[MessageLikeRepresentation],
(...)
[943](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:943) **kwargs: Any,
[944](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:944) ) -> None:
[945](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:945) """Create a chat prompt template from a variety of message formats.
[946](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:946)
[947](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:947) Args:
(...)
[990](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:990)
[991](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:991) """
--> [992](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:992) _messages = [
[993](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:993) _convert_to_message(message, template_format) for message in messages
[994](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:994) ]
[996](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:996) # Automatically infer input variables from messages
[997](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:997) input_vars: set[str] = set()
File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:993, in <listcomp>(.0)
[938](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:938) def __init__(
[939](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:939) self,
[940](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:940) messages: Sequence[MessageLikeRepresentation],
(...)
[943](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:943) **kwargs: Any,
[944](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:944) ) -> None:
[945](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:945) """Create a chat prompt template from a variety of message formats.
[946](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:946)
[947](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:947) Args:
(...)
[990](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:990)
[991](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:991) """
[992](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:992) _messages = [
--> [993](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:993) _convert_to_message(message, template_format) for message in messages
[994](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:994) ]
[996](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:996) # Automatically infer input variables from messages
[997](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:997) input_vars: set[str] = set()
File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:1454, in _convert_to_message(message, template_format)
[1452](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1452) message_type_str, template = message
[1453](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1453) if isinstance(message_type_str, str):
-> [1454](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1454) _message = _create_template_from_message_type(
[1455](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1455) message_type_str, template, template_format=template_format
[1456](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1456) )
[1457](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1457) else:
[1458](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1458) _message = message_type_str(
[1459](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1459) prompt=PromptTemplate.from_template(
[1460](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1460) cast(str, template), template_format=template_format
[1461](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1461) )
[1462](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1462) )
File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:1365, in _create_template_from_message_type(message_type, template, template_format)
[1351](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1351) """Create a message prompt template from a message type and template string.
[1352](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1352)
[1353](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1353) Args:
(...)
[1362](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1362) ValueError: If unexpected message type.
[1363](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1363) """
[1364](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1364) if message_type in ("human", "user"):
-> [1365](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1365) message: BaseMessagePromptTemplate = HumanMessagePromptTemplate.from_template(
[1366](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1366) template, template_format=template_format
[1367](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1367) )
[1368](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1368) elif message_type in ("ai", "assistant"):
[1369](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1369) message = AIMessagePromptTemplate.from_template(
[1370](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1370) cast(str, template), template_format=template_format
[1371](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:1371) )
File c:\Users\alan-\miniconda3\Lib\site-packages\langchain_core\prompts\chat.py:565, in _StringImageMessagePromptTemplate.from_template(cls, template, template_format, partial_variables, **kwargs)
[563](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:563) prompt.append(img_template_obj)
[564](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:564) else:
--> [565](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:565) raise ValueError(f"Invalid template: {tmpl}")
[566](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:566) return cls(prompt=prompt, **kwargs)
[567](file:///C:/Users/alan-/miniconda3/Lib/site-packages/langchain_core/prompts/chat.py:567) else:
ValueError: Invalid template: {'type': 'media', 'mime_type': 'application/pdf', 'data':
Description
I'm trying to send a PDF file as bytes to Gemini using the ChatPromptTemplate method.
Sending the PDF to Gemini seems to works fine as per #4589 and #215
But when using the ChatPromptTemplate, it appears that the code here is not ready to deal with PDFs type in the prompt template.
System Info
langchain==0.3.2 langchain-community==0.3.1 langchain-core==0.3.9 langchain-google-genai==2.0.0 langchain-text-splitters==0.3.0 langgraph==0.2.34 langgraph-checkpoint==2.0.0 langsmith==0.1.131
