pydantic
Support referencing file_ids previously uploaded to model providers.
Description
OpenAI supports uploading files to its files api, which can then be referenced by file_id when calling the inference endpoints that accept the messages property:
Some other providers such as Google support similar APIs :https://ai.google.dev/api/files
This feature is useful because it avoids having to re-upload images or documents in multi-turn conversations.
Looking at the _map_user_prompt function, it is clear that PydanticAI doesn't currently support this feature and provides no way to use it (such as allowing custom message content to be added when calling inference endpoint).
I would like to contribute this to PydanticAI and have two questions:
- Would this be a desirable contribution?
- What would be the recommended strategy to support this in a cross-provider way? My first instinct is to add a new UserContent type (FileId, or FileReference) which would then be handled in
_map_user_promptfor OpenAIModel.
References
No response
@tarruda With Google, uploaded files are already supported as it returns a file URL that can be used with ImageUrl (etc) as of https://github.com/pydantic/pydantic-ai/pull/2270, but you're right that for that for OpenAI and Anthropic (https://docs.anthropic.com/en/docs/build-with-claude/files#using-a-file-in-messages) we'd need a new object that takes a file_id. We can also make it work for Google when the file URL is provided as file_id.
I would indeed need to be a new type listed under UserContent. What do you think about UploadedFile? We could also (later, not necessarily in this PR) add an upload_file method to Provider that returns an object of that type.
A contribution would be more than welcome, we'd just want it implemented for the big three providers to make sure the abstraction is strong enough. Some of them may require mime types, for example.
This would be an awesome feature to have, as it would reduce the latency and size of requests especially for long interactions with multiple files!
@bitnahian I've reopened https://github.com/pydantic/pydantic-ai/pull/2611 because @tarruda is going to continue working on it :)