vercel
ai sdk useChat experiemental_attachment with message
Description
const {
messages,
isLoading: isGenerating,
append: generateDescription,
error: generationError,
setMessages
} = useChat({
sendExtraMessageFields: true,
api: '/api/generate-listing-parts',
id: section.id,
body: {
projectId: project_id,
sectionId: section.id
},
onFinish: (message, {usage, finishReason}) => {
console.log('Finished streaming message:', message)
console.log('Token usage:', usage)
console.log('Finish reason:', finishReason)
},
onError: error => {
console.error('An error occurred:', error)
},
onResponse: response => {
console.log('Received HTTP response from server:', response)
}
})
then later down
await generateDescription({
role: 'user',
content: '',
experimental_attachments: [...photos, ...uploadedPhotos].map(
image => ({
url: image.url,
type: image.type
})
)
})
for some reason the experimental_attachments are not sent
Code example
No response
AI provider
No response
Additional context
No response
The intention was that I can call the following POST api with this data and it would just all work
Similar to the example: https://sdk.vercel.ai/docs/ai-sdk-ui/chatbot#attachments-experimental
Only difference is im not using handleInputChange, handleSubmit
const result = streamText({
abortSignal: request.signal,
model: anthropic('claude-3-5-sonnet-20241022'),
system: SYSTEM_PROMPT,
experimental_transform: smoothStream({chunking: 'word'}),
messages,
})
return result.toDataStreamResponse()
Yep. I experience the same issue. Basically it is not possible to add attachments when using the append. In the handleMessage it works just fine..
Is that right, it doesn't work with append()? I've been banging my head against this for hours. It's documented as a property on ChatRequestOptions in the append() reference.
https://sdk.vercel.ai/docs/reference/ai-sdk-ui/use-chat#append.chat-request-options.experimental_attachments
But whenever I pass it there it gets stripped. I also tried passing it in the message parameter, no luck there either.
I actually seem to have it working - for me - by passing experimental_attachments to both message and options?.
await append({
content: event.detail.text,
role: 'user',
experimental_attachments: attachments,
},{
experimental_attachments: attachments,
body: {
// ...
}
})
Had the same issue.
In my case, just passing experimental_attachments to options was enough:
await append({
content: event.detail.text,
role: 'user',
},{
experimental_attachments: attachments,
})
It’s odd that experimental_attachments is defined in both ChatRequestOptions and Message though, when it seems like it should only be in ChatRequestOptions.
@jstjoe Please could you show your code? It doesn't work for me(((
Seems like one cannot set options through append anymore either. Used to set options with append and reload but now my server side gets "undefined" on additional options. Something happened between AI SDK 3 and 4
The issue is here: https://github.com/vercel/ai/blob/bbcf122be787271de082a40d41b5cb9f5d5eaea8/packages/react/src/use-chat.ts#L403
Even though the message object is the first argument, it only prepares attachments from the chatrequestoptions object, then pipes that back into the message.
@lgrammel ^^
For expo/react native users, you need this polyfill
if (Platform.OS !== "web") {
const setupPolyfills = async () => {
// @ts-ignore
const { polyfillGlobal } = await import("react-native/Libraries/Utilities/PolyfillFunctions");
// ... other polyfills
polyfillGlobal(
"FileList",
() =>
class FileList {
[Symbol.toStringTag]?: string;
constructor() {
this[Symbol.toStringTag] = "FileList";
}
},
);
};
setupPolyfills();
}
实际上，我似乎_通过传递给 和_ 来工作 - 对我来说。
experimental_attachments``message``options?await append({ content: event.detail.text, role: 'user', experimental_attachments: attachments, },{ experimental_attachments: attachments, body: { // ... } })
He solved my problem, it's a bit difficult to describe
@yegor-klymenchuk @martgra can you provide some more information in how you're calling append, we have an E2E test that ensures it should work: https://github.com/vercel/ai/blob/main/packages/react/src/use-chat.ui.test.tsx#L1532-L1547
code examples would help greatly
@yegor-klymenchuk @martgra can you provide some more information in how you're calling
append, we have an E2E test that ensures it should work: https://github.com/vercel/ai/blob/main/packages/react/src/use-chat.ui.test.tsx#L1532-L1547code examples would help greatly
Issue is here: https://github.com/vercel/ai/blob/d866921e5c68ab9113b89d7926456ff66c55101e/packages/ui-utils/src/types.ts#L76
The functionality is right but the types are wrong. The Message interface allows for an experimental_attachments property which is not handled in the append function. The append function only handles experimental_attachments from the options and not the message.
yep I fixed that issue in https://github.com/vercel/ai/pull/6044, but they mentioned that even with the second arg it still didn't work - so would like to know how exactly they're using it