amplify-category-api icon indicating copy to clipboard operation
amplify-category-api copied to clipboard

Add support for Bedrock Prompt Caching in AI Kit

Open su-mskiles opened this issue 8 months ago • 1 comments

Environment information

Not needed.

Describe the feature

AWS recently announced a prompt caching feature for the Converse API that can improve latency and drastically reduce token cost. It would be great if we could optionally enable this when defining a Conversation model.

Use case

To speed things up and save money! I'm personally most interested in enabling caching for the System Prompt since it's defined on the data model, and is always the exact same for every new instance of <AIConversation>. Should be as easy as including an extra "cachePoint" parameter when invoking the Converse API:

...
  "system": [ 
    {
        "text": "You are an app that creates play lists for a radio station that plays rock and pop music. Only return song names and the artist. "
    },
    {
        "cachePoint": {
            "type": "default"
        }
    }
  ],
...

su-mskiles avatar May 02 '25 20:05 su-mskiles