amplify-category-api
amplify-category-api copied to clipboard
Add support for Bedrock Prompt Caching in AI Kit
Environment information
Not needed.
Describe the feature
AWS recently announced a prompt caching feature for the Converse API that can improve latency and drastically reduce token cost. It would be great if we could optionally enable this when defining a Conversation model.
Use case
To speed things up and save money! I'm personally most interested in enabling caching for the System Prompt since it's defined on the data model, and is always the exact same for every new instance of <AIConversation>. Should be as easy as including an extra "cachePoint" parameter when invoking the Converse API:
...
"system": [
{
"text": "You are an app that creates play lists for a radio station that plays rock and pop music. Only return song names and the artist. "
},
{
"cachePoint": {
"type": "default"
}
}
],
...